[DeTomaso] Archive searching

Himes, Terry (397C) terry.himes at jpl.nasa.gov
Thu May 26 15:41:29 EDT 2016 

Ouuuuu!  Nice.  Works good.  And fast.  Nice job.  !!!!


"A Purple Heart proves you were smart enough to hatch a plan,
 stupid enough to try it and lucky enough to survive!"

Terry W. Himes
JPL Jet Propulsion Laboratory
Dawn Spacecraft Team
Rosetta Sequence Team Lead
Phone: (818) 393-6261
Cell:     (818) 653-8213
Fax:     (818) 393-3147
thimes at jpl.nasa.gov<mailto:thimes at jpl.nasa.gov>
🇺🇸

From: Kurt Byrnes <kdb at kbyrnes.com<mailto:kdb at kbyrnes.com>>
Date: Thursday, May 26, 2016 11:34 AM
To: Terry Himes <Terry.Himes at jpl.nasa.gov<mailto:Terry.Himes at jpl.nasa.gov>>, "DeTomaso at server.detomasolist.com<mailto:DeTomaso at server.detomasolist.com>" <DeTomaso at server.detomasolist.com<mailto:DeTomaso at server.detomasolist.com>>
Subject: Re: [DeTomaso] Archive searching

Terry,

Ha ha!  We're using pipermail for archive storage.  I did a quick look at Elasticsearch, interesting product, java based.  I think it may be overkill for our needs.

I just installed swish-e and it seems to be working somewhat.  I need to make some changes so modified date is posted date and include the OLD archives.  Also, I have not put a cron job in yet to index periodically so for the moment, it's searchable through about 2pm EDT.  I'll need to modify the source to clean up the landing page and output search help stuff but for now, everyone should be able to sort of search the archives at:

http://server.detomasolist.com/mailman/swish.cgi

Let me know if this is a solution everyone likes.

Kurt

On 5/26/2016 11:36 AM, Himes, Terry (397C) wrote:
Kurt,

Putting your personal life before others!!  How dare you!!   ;-))

I know nothing about "mailman" (sorry) but I am working with Docker Containers and for Curiosity we
process (ingest) about 1.2TB of telemetry per week.  It takes Elasticsearch less than a second to query
the entire data-store.  Both Elasticsearch and Dockers are free (I think. I'll check on it) so maybe we can
run a local copy.  AWS and the cloud are not cheap.

Oh!  I just googled mailman.  Interesting.  Mostly written in Python.  Awesome.  Do you use Pipermail or
HyperKitty for archival storage??


Terry




"A Purple Heart proves you were smart enough to hatch a plan,
 stupid enough to try it and lucky enough to survive!"

Terry W. Himes
JPL Jet Propulsion Laboratory
Dawn Spacecraft Team
Rosetta Sequence Team Lead
Phone: (818) 393-6261
Cell:     (818) 653-8213
Fax:     (818) 393-3147
<mailto:thimes at jpl.nasa.gov>thimes at jpl.nasa.gov<mailto:thimes at jpl.nasa.gov>
🇺🇸

From: DeTomaso <<mailto:detomaso-bounces at server.detomasolist.com>detomaso-bounces at server.detomasolist.com<mailto:detomaso-bounces at server.detomasolist.com>> on behalf of Kurt Byrnes <kdb at kbyrnes.com<mailto:kdb at kbyrnes.com>>
Date: Thursday, May 26, 2016 6:43 AM
To: "<mailto:DeTomaso at server.detomasolist.com>DeTomaso at server.detomasolist.com<mailto:DeTomaso at server.detomasolist.com>" <DeTomaso at server.detomasolist.com<mailto:DeTomaso at server.detomasolist.com>>
Subject: [DeTomaso] Archive searching

I've started looking into providing search capabilities to the
archives.  There are several utilities that have been tied into mailman
that I'll look into.


I'll get things setup so google can crawl the archives however I think
they only crawl once a week so recent data wouldn't be available.


I'm a bit swamped with real work so please be patient.

_______________________________________________


Detomaso Forum NO LONGER Managed by POCA
Posted emails must not exceed 1.5 Megabytes
DeTomaso mailing list
DeTomaso at server.detomasolist.com<mailto:DeTomaso at server.detomasolist.com>
http://server.detomasolist.com/mailman/listinfo/detomaso

To manage your subscription (change email address, unsubscribe, etc.) use the links above.

Members who post to this list grant license to the list to forward any message posted here to all past, current, or future members of the list. They also grant the list owner permission to maintain an archive or approve the archiving of list messages.


-------------- next part --------------
   Ouuuuu!  Nice.  Works good.  And fast.  Nice job.  !!!!

   "A Purple Heart proves you were smart enough to hatch a plan,

    stupid enough to try it and lucky enough to survive!"

   Terry W. Himes

   JPL Jet Propulsion Laboratory

   Dawn Spacecraft Team

   Rosetta Sequence Team Lead

   Phone: (818) 393-6261

   Cell:     (818) 653-8213

   Fax:     (818) 393-3147

   [1]thimes at jpl.nasa.gov

   From: Kurt Byrnes <[2]kdb at kbyrnes.com>
   Date: Thursday, May 26, 2016 11:34 AM
   To: Terry Himes <[3]Terry.Himes at jpl.nasa.gov>,
   "[4]DeTomaso at server.detomasolist.com"
   <[5]DeTomaso at server.detomasolist.com>
   Subject: Re: [DeTomaso] Archive searching

   Terry,
   Ha ha!  We're using pipermail for archive storage.  I did a quick look
   at Elasticsearch, interesting product, java based.  I think it may be
   overkill for our needs.
   I just installed swish-e and it seems to be working somewhat.  I need
   to make some changes so modified date is posted date and include the
   OLD archives.  Also, I have not put a cron job in yet to index
   periodically so for the moment, it's searchable through about 2pm EDT.
   I'll need to modify the source to clean up the landing page and output
   search help stuff but for now, everyone should be able to sort of
   search the archives at:
   [6]http://server.detomasolist.com/mailman/swish.cgi
   Let me know if this is a solution everyone likes.
   Kurt
   On 5/26/2016 11:36 AM, Himes, Terry (397C) wrote:

   Kurt,
   Putting your personal life before others!!  How dare you!!   ;-))
   I know nothing about "mailman" (sorry) but I am working with Docker
   Containers and for Curiosity we
   process (ingest) about 1.2TB of telemetry per week.  It takes
   Elasticsearch less than a second to query
   the entire data-store.  Both Elasticsearch and Dockers are free (I
   think. I'll check on it) so maybe we can
   run a local copy.  AWS and the cloud are not cheap.
   Oh!  I just googled mailman.  Interesting.  Mostly written in Python.
   Awesome.  Do you use Pipermail or
   HyperKitty for archival storage??
   Terry

   "A Purple Heart proves you were smart enough to hatch a plan,

    stupid enough to try it and lucky enough to survive!"

   Terry W. Himes

   JPL Jet Propulsion Laboratory

   Dawn Spacecraft Team

   Rosetta Sequence Team Lead

   Phone: (818) 393-6261

   Cell:     (818) 653-8213

   Fax:     (818) 393-3147

   [7]thimes at jpl.nasa.gov

   From: DeTomaso <[8]detomaso-bounces at server.detomasolist.com> on behalf
   of Kurt Byrnes <[9]kdb at kbyrnes.com>
   Date: Thursday, May 26, 2016 6:43 AM
   To: "[10]DeTomaso at server.detomasolist.com"
   <[11]DeTomaso at server.detomasolist.com>
   Subject: [DeTomaso] Archive searching

   I've started looking into providing search capabilities to the
   archives.  There are several utilities that have been tied into mailman
   that I'll look into.
   I'll get things setup so google can crawl the archives however I think
   they only crawl once a week so recent data wouldn't be available.
   I'm a bit swamped with real work so please be patient.
   _______________________________________________
   Detomaso Forum NO LONGER Managed by POCA
   Posted emails must not exceed 1.5 Megabytes
   DeTomaso mailing list
   [12]DeTomaso at server.detomasolist.com
   [13]http://server.detomasolist.com/mailman/listinfo/detomaso
   To manage your subscription (change email address, unsubscribe, etc.)
   use the links above.
   Members who post to this list grant license to the list to forward any
   message posted here to all past, current, or future members of the
   list. They also grant the list owner permission to maintain an archive
   or approve the archiving of list messages.

References

   Visible links
   1. mailto:thimes at jpl.nasa.gov
   2. mailto:kdb at kbyrnes.com
   3. mailto:Terry.Himes at jpl.nasa.gov
   4. mailto:DeTomaso at server.detomasolist.com
   5. mailto:DeTomaso at server.detomasolist.com
   6. http://server.detomasolist.com/mailman/swish.cgi
   7. mailto:thimes at jpl.nasa.gov
   8. mailto:detomaso-bounces at server.detomasolist.com
   9. mailto:kdb at kbyrnes.com
  10. mailto:DeTomaso at server.detomasolist.com
  11. mailto:DeTomaso at server.detomasolist.com
  12. mailto:DeTomaso at server.detomasolist.com
  13. http://server.detomasolist.com/mailman/listinfo/detomaso

   Hidden links:
  15. mailto:thimes at jpl.nasa.gov
  16. mailto:detomaso-bounces at server.detomasolist.com
  17. mailto:DeTomaso at server.detomasolist.com

More information about the DeTomaso mailing list