Slack Export - #dagster-integration

Dalin Kim (dalinkim@northwesternmutual.com)

2022-01-21 21:26:13

@Dalin Kim has joined the channel

Kevin Mellott (kevin.r.mellott@gmail.com)

2022-01-21 21:28:55

@Kevin Mellott has joined the channel

Nafisah Islam (nafisahislam@northwesternmutual.com)

2022-01-21 21:28:56

@Nafisah Islam has joined the channel

Antonio Moctezuma (antoniomoctezuma@northwesternmutual.com)

2022-01-21 21:28:56

@Antonio Moctezuma has joined the channel

Joshua Wankowski (joshuawankowski@northwesternmutual.com)

2022-01-21 21:28:56

@Joshua Wankowski has joined the channel

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-01-21 21:28:56

@Maciej Obuchowski has joined the channel

Julien Le Dem (julien@apache.org)

2022-01-21 21:28:56

@Julien Le Dem has joined the channel

Dalin Kim (dalinkim@northwesternmutual.com)

2022-01-21 22:17:56

Hello, my team would like to contribute to the OpenLineage-Dagster integration work and wanted to start a public channel for general discussion on this topic.

Issue #489 is currently open for review and includes the proposal for the integration. As we proceed with the initial implementation, we’d appreciate feedback from the community to make sure the approach is reasonable and OpenLineage events are captured accurately.

Looking forward to more discussions. Thanks!

<https://github.com/OpenLineage/OpenLineage/issues/489|#489 [INTEGRATION][Dagster] Openlineage-Dagster Integration>

Hello, my team would like to contribute to the OpenLineage-Dagster integration work with the following proposal. Integration Target: <a href="https://github.com/dagster-io/dagster">Dagster</a> Proposal: New openlineage-dagster library that integrates Dagster with OpenLineage for automatic metadata collection Incremental Support: <ol><li>Initial integration handling basic metadata (job runs)</li><li>Individual “extractor” handling additional metadata (input/output datasets, data quality metrics, and etc.)</li> </ol> Proposed Implementation: Dagster uses an <a href="https://docs.dagster.io/deployment/executors#executors"><code>Executor</code></a> to <a href="https://docs.dagster.io/_apidocs/internals#dagster.Executor.execute"><code>execute()</code></a> Dagster job (aka pipeline) runs, and each execution yields a stream of <a href="https://docs.dagster.io/_apidocs/execution#dagster.DagsterEvent"><code>DagsterEvents</code></a> from different stages of the run. As Dagster supports custom Executor, we can implement a custom <code>OpenLineageExecutor</code> that will: • act as a thin wrapper over an existing implementation of Executor • translate given <a href="https://docs.dagster.io/_apidocs/internals#dagster.PipelineRun"><code>PipelineRun</code></a> during execution and emit parent job (pipeline/job-level) RunEvents • execute steps using its helper (actual) executor • intercept yielded DagsterEvents, translate, and emit child job (op/step-level) RunEvents • yield original DagsterEvents Similar to openlienage-airflow, Dagster-specific <code>OpenLineageAdapter</code> can be implemented as a helper. Generated OpenLineage Jobs & RunEvents: With the above initial work, the following jobs and events can be generated. Dagster job (pipeline) run -> OpenLineage parent job run • During execution, <code>PipelineRun</code> is available from <code>PlanOrchestrationContext</code>, and we can get job/pipeline-level metadata from this PipelineRun and translate it to an OpenLineage event. • As an example, given <code>pipeline_run</code>, a RunEvent may look like, <pre><code>RunEvent( eventType=event_type, eventTime=datetime.now().isoformat(), run=Run( runId=pipeline_run.run_id, ), job=Job( namespace=namespace, name=pipeline_run.pipeline_name, ), producer=PRODUCER, ) </code></pre> Dagster op (step) run -> OpenLineage child job run • Assuming "op" is equivalent to "step" at Dagster's low-level, we can identify op-related events from the returned DagsterEvents based on its <a href="https://docs.dagster.io/_apidocs/execution#dagster.DagsterEventType"><code>DagsterEventType</code></a>. For the initial implementation, the following types can be handled. • DagsterEventType.STEPSTART • DagsterEventType.STEPSUCCESS • DagsterEventType.STEP_FAILURE • Other types may be usable to extract additional metadata. • As an example, given <code>dagster_event</code>, a RunEvent may look like, <pre><code>RunEvent( eventType=event_type, eventTime=datetime.now().isoformat(), run=self._build_run( namespace=namespace, run_id=run_id, # not associated with Dagster parent_run_id=dagster_event.logging_tags.get("run_id"), # pipeline run id parent_job_name=dagster_event.logging_tags.get("pipeline_name"), ), job=self._build_job( namespace=namespace, job_name=f"{dagster_event.pipeline_name}.{dagster_event.step_key}", ), producer=PRODUCER, )) </code></pre> How to Use in Dagster: OpenLineageExecutor's ExecutorDefinition can be defined at the job-level like below. <pre><code>from dagster import job, op from openlineage.dagster.executor import openlineage_executor @job(executor_def=openlineage_executor) def a_job(): @op def an_op(): pass an_op() </code></pre> And the executor is configurable like below as Dagster's "executor config". <pre><code>execution: config: openlienage: url: "https://..." api_key: "" namespace: "default" executor: multiprocess_executor: max_concurrent: 2 </code></pre> <hr /> This proposal is based on limited research and understanding of Dagster, and there may be better alternatives as well that we are happy to discuss over and learn more so any feedback from the community is appreciated as we lay this initial block. Thank you for reviewing and looking forward to further discussions!

Comments

👏 Eric Veleker

firas (firas.omrane.contact@gmail.com)

2022-01-22 17:08:23

@firas has joined the channel

Laurent Paris (laurent@datakin.com)

2022-01-24 11:49:36

@Laurent Paris has joined the channel

Julien Le Dem (julien@apache.org)

2022-01-24 20:40:19

FYI, I reached out to the Dagster community and they replied to @Dalin Kim’s ticket: https://github.com/OpenLineage/OpenLineage/issues/489#issuecomment-1020718071

<https://github.com/OpenLineage/OpenLineage/issues/489#issuecomment-1020718071|Comment on #489 [INTEGRATION][Dagster] Openlineage-Dagster Integration>

Hi <a href="https://github.com/dalinkim">@dalinkim</a> - I'm a Dagster engineer, here to answer any questions you have. It would be awesome to see a Dagster-OpenLineage integration. A challenge with accomplishing this via a custom executor is that there are already a number of executor implementations that dagster users rely on. E.g. some use the K8s executor, some use the in-process executor, and some use the multiprocess executor. If you wanted to maintain that flexibility and also integrate via the executor, you'd need to write an openlineage version of each of those executors. The way that we've recommended implementing similar integrations in the past is via a Dagster sensor that tails the Dagster event log. Something like: <pre><code>from dagster import sensor from dagster.core.storage.event_log.base import EventRecordsFilter @sensor def openlineage_sensor(context): event_records = context.instance.get_event_records( EventRecordsFilter(after_cursor=context.cursor), ascending=True, limit=5, ) for event_record in event_records: event_log_entry = event_record.event_log_entry storage_id = event_record.storage_id # write relevant contents of the event log entry to OpenLineage context.update_cursor(storage_id) </code></pre>

Julien Le Dem (julien@apache.org)

2022-01-24 20:40:41

Thank you for getting this going @Dalin Kim!

Michael Robinson (michael.robinson@astronomer.io)

2022-02-04 15:53:24

@Michael Robinson has joined the channel

👋 Michael Robinson

Julien Le Dem (julien@apache.org)

2022-02-04 15:59:17

Let me intro @Michael Robinson who among other things is looking for topics to speak about in the OpenLineage monthly meeting

Michael Robinson (michael.robinson@astronomer.io)

2022-02-04 16:08:51

Thanks, @Julien Le Dem. If anyone is interested in speaking at the next OL TSC meeting about their work on the Dagster integration, please reply here or message me. The integration is on the agenda for the upcoming https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting|meeting on 2/9 at 9 am PT. @Dalin Kim

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-04 17:18:32

*Thread Reply:* Hi Michael, While I’m currently going through internal review process before creating a PR, I can do a quick demo on the current OpenLineage sensor approach and get some initial feedback if that is okay.

Julien Le Dem (julien@apache.org)

2022-02-09 13:04:24

*Thread Reply:* thanks for the demo!

👍 Dalin Kim

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-10 22:37:06

*Thread Reply:* Thanks for the opportunity!

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-15 13:53:17

Hello, I created a pull request for the OpenLineage sensor work here. This initial work handles the basic lifecycles of Dagster jobs & ops (pipelines & steps), and more discussion will be needed to define how we can handle datasets. Hopefully, this PR is acceptable as an initial groundwork, and all feedback is appreciated. Thanks!

<https://github.com/OpenLineage/OpenLineage/pull/541|#541 [INTEGRATION][Dagster] OpenLineage sensor>

Signed-off-by: Dalin Kim <a href="mailto:dalinkim@northwesternmutual.com">dalinkim@northwesternmutual.com</a> Problem OpenLineage-Dagster integration is not available. Closes: <a href="https://github.com/OpenLineage/OpenLineage/issues/489">#489</a> Solution Introduce an OpenLineage sensor that tails Dagster event logs for tracking metadata. On each sensor evaluation, the function processes a batch of event logs, converts Dagster events into OpenLineage events, and emits them to an OpenLineage backend. Dagster job is converted to an OpenLineage parent job with its individual steps converted to children jobs. Event type conversion: > Note: All schema changes require discussion. Please <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue">link the issue</a> for context. ☐ Your change modifies the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.json">core</a> OpenLineage model ☐ Your change modifies one or more OpenLineage <a href="https://github.com/OpenLineage/OpenLineage/tree/main/spec/facets">facets</a> If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports <code>S3</code> and <code>GCS</code> filesystem operations, tested with AWS EMR). Checklist ☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work ☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> ☑︎ Your changes are accompanied by tests (if relevant) ☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained ☑︎ You've updated any relevant documentation (if relevant) ☑︎ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary) ☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (if relevant)

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-02-15 15:39:14

*Thread Reply:* Thanks for the PR! I'll take a look tomorrow.

👍 Dalin Kim

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-02-16 09:37:27

*Thread Reply:* @Dalin Kim looks great! I approved it. One thing to do is to rebase on main and force-with-lease push: it appears that there are some conflicts.

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-16 11:59:49

*Thread Reply:* @Maciej Obuchowski Thank you for the review. Just had a small conflict in CHANGELOG, which has been resolved. For integration test, do you suggest a similar approach like airflow integration using flask?

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-16 12:03:08

*Thread Reply:* Also, I reached out to Sandy from Dagster for a review to make sure this is good from Dagster side of things.

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-02-16 12:09:12

*Thread Reply:* > For integration test, do you suggest a similar approach like airflow integration using flask? Yes, I think we can reuse that part. The most important thing is that we have real Dagster running "real" workloads.

👍 Dalin Kim

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-17 00:45:37

*Thread Reply:* Documentation has been updated based on feedback from Yuhan from Dagster team, and I believe everything is set from my end.

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-02-17 05:25:16

*Thread Reply:* Great! Let's just get rid of this linting error and I'll merge it then.

https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/2204/workflows/cdb412e6-b41a-4fab-bc8e-d8bee71d051d/jobs/18963

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-17 11:27:41

*Thread Reply:* Thank you. I overlooked this when updating docstring. All should be good now.

One final question - should we make the dagster unit test job “required” in the ci and how can that be configured?

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-02-17 11:54:08

*Thread Reply:* @Willy Lulciuc I think you configured it, am I right?

@Dalin Kim one more rebase, please 🙏 I've turned auto-merge on, but unfortunately I can't rebase your branch on fork.

👍 Dalin Kim

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-17 11:58:34

*Thread Reply:* Rebased and pushed

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-02-17 12:23:44

*Thread Reply:* @Dalin Kim merged!

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-17 12:24:28

*Thread Reply:* @Maciej Obuchowski Awesome! Thank you so much for all your help!

🙌 Maciej Obuchowski

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-15 14:01:43

One small question on ci - airflow integration test checks are stuck in the “expected” state. Is this expected or is there something I missed in the ci update?

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-02-15 14:47:49

*Thread Reply:* Sorry - we're not running integration tests for airflow on forks due to security reason. If you're not touching any Airflow files then it should not affect you at all 🙂

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-02-15 14:48:23

*Thread Reply:* In other words, it's just as expected.

Dalin Kim (dalinkim@northwesternmutual.com)

2022-02-15 15:09:42

*Thread Reply:* Thank you for the clarification

Willy Lulciuc (willy@datakin.com)

2022-02-17 11:54:18

@Willy Lulciuc has joined the channel

Nicola Monger (nicola.monger@moonpig.com)

2022-02-17 16:52:44

@Nicola Monger has joined the channel

John Thomas (john@datakin.com)

2022-02-18 13:03:29

@John Thomas has joined the channel

Dominique Tipton (dominiquetipton@northwesternmutual.com)

2022-03-01 17:21:49

@Dominique Tipton has joined the channel

Dominique Tipton (dominiquetipton@northwesternmutual.com)

2022-03-01 17:36:02

Hi all 👋

I have opened up an issue/proposal on getting datasets incorporated with the dagster integration. I would love to get some feedback and conversations going with the community on the proposed approach. Thanks!

<https://github.com/OpenLineage/OpenLineage/issues/583|#583 [INTEGRATION][DAGSTER] Add capability to include input and output datasets with the run event>

With the initial work that has been done to integrate OpenLineage with Dagster (thanks <a href="https://github.com/dalinkim">@dalinkim</a>!), it seems like the next step should be to include a way to emit datasets along with the run events. Proposal: Introduce a way to transform Dagster assets or Outputs into InputDatasets and/or OutputDatasets to improve metadata collection. Proposed Implementation: I am no Dagster expert, but after working with the assets for a bit, there unfortunately doesn't seem to be a good way to automatically transform a generic <a href="https://docs.dagster.io/_apidocs/ops#dagster.AssetMaterialization">AssetMaterialization</a> or <a href="https://docs.dagster.io/_apidocs/ops#dagster.Output">Output</a> into an OpenLineage Dataset. However, specific OpenLineage assets can be manually created and yielded in a Dagster job/pipeline. Through the use of a wrapper/asset-generating function, a sufficiently unique AssetMaterialization with a defined structure can be yielded and later parsed into a dataset and added to the run event. > Through the use of a wrapper/asset-generating function, a sufficiently unique AssetMaterialization with a defined structure can be created/yielded This function would take in an input/output dataset and create an AssetMaterialization—with an <a href="https://docs.dagster.io/_apidocs/ops#dagster.AssetKey">AssetKey</a> or unique structure that can be used to identify the asset as an OpenLineage dataset. That asset could then be yielded in a Dagster job/pipeline. > and later parsed into a dataset An extra <code>elif</code> check can be added to the <a href="https://github.com/OpenLineage/OpenLineage/blob/8c15b9fe9dc88799d44fc3999f18a47909acb844/integration/dagster/openlineage/dagster/sensor.py#L145-L157">OpenLineage-Dagster sensor</a> to detect <a href="https://docs.dagster.io/_apidocs/execution#dagster.DagsterEventType.ASSET_MATERIALIZATION">when assets are yielded</a>. The asset can then be determined if it is a dataset by its AssetKey/structure and if it is, it can be parsed back into an input/output dataset by reversing the logic used in the above function. > and added to the run event After the datasets have been parsed and are available in the sensor, they can be passed along to the emit methods so they can be added to the run event. Example: To emit a dataset, something similar to this would be added to a job/pipeline: <pre><code>@op def an_op(): output_dataset = OutputDataset( namespace="example_namespace", name="example_name" ) asset = asset_generating_function(output_dataset) yield asset # or yield it directly in the function # asset_yielding_function(output_dataset) </code></pre> Limitations of this approach: • In order to fit with other assets yielded by Dagster, the AssetKey should be 100% human readable (i.e. no UUIDs and have a hierarchy that makes sense). • The created AssetKey should be idempotent. A dataset with the same namespace/name should produce the same key so dagster can relate the assets to one another. A dataset is identified by its namespace and name and the asset should be too. • The AssetMaterialization (most likely the AssetKey) should contain a sufficiently unique identifier so other assets are not mistakenly identified as an OpenLineage dataset. • A good option for the AssetKey limitations could be ["OpenLineage", "InputDataset OR OutputDataset OR Dataset", "<dataset.namespace>", "<dataset.name>"] This approach is human readable, idempotent, and is a sufficiently unique identifier (assuming other assets beginning with OpenLineage > Dataset are not being created elsewhere).

👍 Dalin Kim, Maciej Obuchowski

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-03-04 06:59:34

FYI @Dalin Kim @Dominique Tipton Dagster 0.14.3 broke something, and unit tests started to fail. I've pinned version to 0.14.2 for now, but can you take a look?

Dalin Kim (dalinkim@northwesternmutual.com)

2022-03-04 09:21:58

*Thread Reply:* Thanks for letting us know. We’ll take a look and follow up.

Dalin Kim (dalinkim@northwesternmutual.com)

2022-03-04 12:32:26

*Thread Reply:* Just as an update on findings, it appears that this MR introduced a breaking change for the test helper function that creates a test EventLogRecord.

<https://github.com/dagster-io/dagster/pull/6769|#6769 [events] remove message field>

<https://github.com/OpenLineage/OpenLineage/blob/main/integration/dagster/tests/conftest.py | conftest.py>

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-03-04 06:59:46

test log: https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/2424/workflows/a426[…]5c1-b2f1-3a8e0cf61bd9/jobs/21719/parallel-runs/0/steps/0-105

👀 Dalin Kim, Dominique Tipton

Dalin Kim (dalinkim@northwesternmutual.com)

2022-03-04 14:38:59

Here is PR to fix the failing tests with latest Dagster version.

<https://github.com/OpenLineage/OpenLineage/pull/588|#588 [INTEGRATION][Dagster] Handle updated EventLogEntry in OpenLineage sensor unit test>

Problem In Dagster 0.14.3, <code>message</code> field was removed from EventLogEntry per <a href="https://github.com/dagster-io/dagster/pull/6769">dagster-io/dagster#6769</a>. This is Dagster's internal change that should not affect the functionality of the OpenLineage sensor, but the unit test helper function that creates a mock EventLogEntry needs an update to handle this differently based on Dagster version. Solution Update unit test to handle removed <code>message</code> field in EventLogEntry based on Dagster version Checklist ☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work ☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> ☑︎ Your changes are accompanied by tests (if relevant) ☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained ☐ You've updated any relevant documentation (if relevant) ☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary) ☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (if relevant)

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-03-04 15:35:01

Thanks! Merged.

Ofek Braunstein (ofekbraunshtein@gmail.com)

2022-03-06 12:35:36

@Ofek Braunstein has joined the channel

David ROBERT (david.robert.ext@louisvuitton.com)

2022-03-18 11:19:18

@David ROBERT has joined the channel

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-03-21 11:11:09

Hello, Dagster tests are failing on main: https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/2675/workflows/fdaf[…]704-ad9b-e33d2b1e956c/jobs/24743/parallel-runs/0/steps/0-105

I think something broke again with 0.14.5

Dominique Tipton (dominiquetipton@northwesternmutual.com)

2022-03-21 12:56:33

*Thread Reply:* Thanks for the heads up. We’ll look into it and follow up

Dominique Tipton (dominiquetipton@northwesternmutual.com)

2022-03-22 16:02:01

*Thread Reply:* Here is the PR to fix the error with the latest Dagster version

<https://github.com/OpenLineage/OpenLineage/pull/624|#624 [INTEGRATION][Dagster] Handle updated PipelineRun in OpenLineage sensor unit test>

Problem In Dagster 0.14.4, <code>pipeline_name</code> was changed from an optional argument to a required argument of PipelineRun per <a href="https://github.com/dagster-io/dagster/pull/6917">dagster-io/dagster#6917</a>. This is Dagster's internal change that should not affect the functionality of the OpenLineage sensor, but the unit test helper function that creates a PipelineRun needs an update to include the required argument. Solution Update unit test helper method to include the required <code>pipeline_name</code> argument in PipelineRun. Checklist ☑︎ You've <a href="https://github.com/OpenLineage/OpenLineage/blob/main/why-the-dco.md">signed-off</a> your work ☑︎ Your pull request title follows our <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md#creating-pull-requests">guidelines</a> ☑︎ Your changes are accompanied by tests (if relevant) ☑︎ Your change contains a <a href="https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6">small diff</a> and is self-contained ☐ You've updated any relevant documentation (if relevant) ☐ You've updated the <a href="https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md"><code>CHANGELOG.md</code></a> with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary) ☐ You've versioned the core OpenLineage model or facets according to <a href="https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver">SchemaVer</a> (if relevant)

Comments

Maciej Obuchowski (maciej.obuchowski@getindata.com)

2022-03-23 08:20:41

*Thread Reply:* Thanks! Merged.

Dominique Tipton (dominiquetipton@northwesternmutual.com)

2022-03-23 09:49:53

*Thread Reply:* Thanks!

marc_pan (pxy0592@gmail.com)

2022-03-28 23:03:45

@marc_pan has joined the channel

Nico Ritschel (nico@antmoney.com)

2022-03-30 20:23:13

@Nico Ritschel has joined the channel

Orbit

2022-03-31 13:21:47

@Orbit has joined the channel

Nico Ritschel (nico@antmoney.com)

2022-03-31 14:18:06

I love the pattern of parsing logs in this integration, so much more flexible compared to the Airflow integration

John Thomas (john@datakin.com)

2022-03-31 15:59:33

*Thread Reply:* That's definitely the advantage of the log-parsing method. The Airflow integration, especially the most recent version for Airflow 2.3+, has the advantage of being more robust when it comes to delivering lineage in real-time

🙌 Nico Ritschel

Nico Ritschel (nico@antmoney.com)

2022-03-31 19:18:29

*Thread Reply:* Thanks for the heads up on this integration!

Nico Ritschel (nico@antmoney.com)

2022-03-31 19:20:10

*Thread Reply:* I suspect future integrations will move towards this pattern as well? Sorry, off-topic for this channel, but this was the first place I've seen this metadata collection method in this project.

Nico Ritschel (nico@antmoney.com)

2022-03-31 19:22:14

*Thread Reply:* I would be curious to explore similar external executor integrations for Airflow, say for Papermill or Kubernetes (via the corresponding operators). Suppose one would need to pass job metadata through to the respective platform where logs are actually collected.

Sudhir Rao (sudhir@zemosolabs.com)

2022-04-01 13:25:58

@Sudhir Rao has joined the channel

Fraser Marlow (fraser@elementl.com)

2023-12-02 21:43:01

On Tuesday, the Dagster team will be showcasing Embedded ELT, a way to save tens of thousands of dollars on data movement tasks. Come catch the session with @Pedram (Dagster) https://dagster.io/events/embedded-elt-dec-2023

dagster.io

Dagster Embedded ELT: You might not need Fivetran

Join Pedram Navid as we explore the capabilities of Dagster's embedded ELT.

Original URL: https://dagster.io/events/embedded-elt-dec-2023

Public Channels

Private Channels

Direct Messages

Group Direct Messages