If you are like me and spend much time in Node-RED but still don't want to lose touch with the outside world, then this flow might be for you!
What that flow does is to post all new forum posts and new topics into your debug panel of Node-RED. So that amongst all the other debug message, new forum posts come on top of that - hoorray!
So yes, it is possible to have new forum posts displayed in Node-RED.
I was playing around with DuckDB WASM and wanted a use-case for testing how to integrate DuckDB into the browser with Node-RED on the server. So I decided to use the Discourse API to load all users of a forum into DuckDB. From there I thought about getting updates of the forum directly into Node-RED.
Of course this is not the only way to achieve this goal, more details about why I did it this way over at the flow page.
I'm no expert but my understanding is that DuckDB is an ephemeral datastore so it is not designed for long term data storage. Apparently it can be configured to store data to disk but it shines being used as an in-memory datastore - similar to SQLite (errr... SQLite has a clear path to on-disk storage, so it differs there).
My understanding is that its core use case is to transform and shape data before that data is stored in some permanent datastore. For example, obtaining unstructured data in JSON format from some API, store that into a DuckDB, structure (i.e. from object to table format) and cleanse the data (e.g. removing corrupt data points) in DuckDB and then export that data from DuckDB (using something like Parquet), to then final import the data into on-disk datastore, e.g, postgres or similar.
I've been playing around with the WASM version to demonstrate how to use in-memory in-browser storage to manipulate data purely in the client, no server required. But unfortunately the WASM api differs much from the on-server in-memory DuckDB API which makes difficult to create something that is truly useful without having DuckDB installed on the server (which is "simple" if not using the Node-RED docker image ..)
What I would finding interesting is to create an orchestration Node-RED flow which would load data into a DuckDB, transform & cleanse that data in DuckDB and then transfer the data to a aggregation datastore - that would be a useful use-case rather than integrating DuckDB into a client side Node-RED instance. The DuckDB Node-RED nodes actually has already created this orchestration use-case, so that's definitely doable.
DuckDB is designed to support analytical query workloads, also known as online analytical processing (OLAP). These workloads are characterized by complex, relatively long-running queries that process significant portions of the stored dataset, for example aggregations over entire tables or joins between several large tables. Changes to the data are expected to be rather large-scale as well, with several rows being appended, or large portions of tables being changed or added at the same time.
