• Maxwell's daemon for MySQL.What about Ingres?

    From nikosv@21:1/5 to All on Wed Aug 5 23:45:01 2020
    Is there something like this

    https://maxwells-daemon.io/#:~:text=Maxwell's%20Daemon&text=This%20is%20Maxwell's%20daemon%2C%20an,a%20place%20to%20write%20to.

    for Ingres?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From nikosv@21:1/5 to All on Thu Aug 6 01:49:13 2020
    Currently I'm using rules and database events as described in : https://www.i-programmer.info/programming/perl/12299-connecting-the-database-to-the-outside-world-with-perl-and-database-events.html
    but I want to rewrite it with a tool like Maxwell, which would give the possibility to treat the data as a stream and feed Kafka.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Roy Hann@21:1/5 to nikosv on Thu Aug 6 08:42:52 2020
    nikosv wrote:

    Is there something like this

    https://maxwells-daemon.io for Ingres?

    If you want something free/open-source, the short answer no.

    Longer answer: people have been crafting their own solutions for as long
    as I can remember.

    In the Ingres world people have used the output of auditdb to generate
    scripts in difference formats.

    Another possibility is to analyze the transaction log file in the way
    HVR does it. HVR is extremely versatile and fast, but it costs money. (https://www.hvr-software.com)

    Neither of these is quite the same as using a MySQL binlog though.
    The MySQL log is a sort of hybrid between the Ingres journal that
    records what did happen, and the Ingres session logs that record what
    was requested. If you try to delete rows that don't exist there
    will be no record of the attempt in the Ingres journals (and hence
    none in the audit output).

    To see everything, including a DELETE that does nothing, you'd need
    to capture and analyze the session logs. Again, people have crafted
    their own tools for that. The log file format used to be fairly well
    documented on the Actian wiki but I don't know where you'd look today.


    Roy

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Roy Hann@21:1/5 to nikosv on Thu Aug 6 09:22:24 2020
    nikosv wrote:

    Currently I'm using rules and database events as described in : https://www.i-programmer.info/programming/perl/12299-connecting-the-database-to-the-outside-world-with-perl-and-database-events.html
    but I want to rewrite it with a tool like Maxwell, which would
    give the possibility to treat the data as a stream and feed Kafka.

    HVR does that.

    https://www.hvr-software.com/wp-content/uploads/2020/07/HVR_Kafka_datasheet.pdf

    Roy

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From nikosv@21:1/5 to All on Thu Aug 6 04:05:03 2020
    this looks promising!

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From nksvg@21:1/5 to nikosv on Thu Aug 6 12:39:46 2020
    On Thursday, August 6, 2020 at 2:05:04 PM UTC+3, nikosv wrote:
    this looks promising!
    but looks pricey...........there's not even a quote on the "Change Data Capture" product pricing!
    I'll have to stick to dbevents I guess...

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)