use @tag on nodes generated with @parameterize_sources

[gravesee]

The @tag decorator is very useful. Is it possible to use on nodes generated with parameterize_sources/parameterize_inputs? Particularly without involving the driver?

[skrawcz]

@gravesee can you clarify by providing some pseudo code and what you would expect to happen?
E.g. do you want each parameterization to have the same tags? or different ones? or?

[skrawcz]

[edit] We have a new decorator @tag_outputs since 1.12.0 that solves this exact case[/edit]

We have two decorators that you can use together:

• @tag
• @tag_outputs

@tag

Tags all the outputs with the same key and value.

@tag_outputs

Tags individual outputs with specific keys and values.

In the example below, we see that @tag is used to set a value for all outputs, while @tag_outputs is only used to add a tag to avg_2wk_spend:

# functions.py
import pandas as pd

from hamilton.function_modifiers import parameterize, tag, tag_outputs
from hamilton.function_modifiers import value, source

@tag_outputs(avg_2wk_spend={"special_tag": "only for avg_2wk_spend"})
@tag(a_tag="this is a tag for all outputs")
@parameterize(
    avg_2wk_spend={'rolling_lookback' : value(2)},
    avg_3wk_spend={'rolling_lookback' : value(3)} 
)
def avg_nwk_spend(spend: pd.Series, rolling_lookback: int) -> pd.Series:
    """Average marketing spend looking back {rolling_lookback} weeks."""
    return spend.rolling(rolling_lookback).mean()

# driver.py
from hamilton import driver
import functions # Keep this names 'functions'
import pandas as pd

# Pass data into the graph externally
inputs = {'spend' : pd.Series([1,2,3,4,5,6])}

# Instantiate a driver 
dr = driver.Driver({}, functions)

all_possible_outputs = dr.list_available_variables()
desired_outputs = [o.name for o in all_possible_outputs
                   if 'this is a tag for all outputs' == o.tags.get('a_tag')]
output = dr.execute(desired_outputs, inputs=inputs)
print(output) # will output both `avg_2wk_spend` and `avg_3wk_spend`.

desired_outputs = [o.name for o in all_possible_outputs
                   if o.tags.get('special_tag')]
output = dr.execute(desired_outputs, inputs=inputs)
print(output) # will output only `avg_2wk_spend`.

Then to prove that they got different tags, you should see two outputs, that differ in the columns output :)

[gravesee]

@skrawcz thank you. I think my users will want more flexibility. I have provided more detail below. I know beggars can't be choosers and hamilton solves a real problem for my team! For data governance we have a lot of metadata for out features. Currently it is either in docstrings or in separate files. I really like how I can keep everything together in hamilton and get to query the metadata.

from hamilton.function_modifiers import parameterize_sources
import pandas as pd


# current code looks like this 
@parameterize_sources(
    **{"public_usage": {"invar": "outvar1"}, "private_usage": {"invar": "outvar2"}}
)
def outvar_logic(invar: pd.Series) -> pd.Series:
    # logic goes here
    return invar

# without parameterize_sources I would have to repeat myself
@tag(usage='public')
def outvar1(...) -> pd.Series:
    ...

@tag(usage='private')
def outvar1(...) -> pd.Series:
    ...

# is there a way to add tags to nodes that are generated? Any programmatic means?
outvar_logic.output_nodes['outvar1'].tags = {
    'usage': 'public' 
}

outvar_logic.output_nodes['outvar2'].tags = {
    'usage': 'private'
}

[skrawcz]

I see. Would the following work -- i.e. split outvar_logic into two? (might be nice from a code/review standpoint to separate them?)

def _any_shared_logic(...) -> ... :
   return ... 

@tag(usage="public")
@parameterize_sources(
    **{"public_usage": {"invar": "outvar1"}}
)
def outvar_logic_public(invar: pd.Series) -> pd.Series:
    # logic goes here -- using `_any_shared_logic()`
    return invar

@tag(usage="private")
@parameterize_sources(
    **{"private_usage": {"invar": "outvar2"}}
)
def outvar_logic_private(invar: pd.Series) -> pd.Series:
   # logic goes here -- using `_any_shared_logic()`
   return invar

is there a way to add tags to nodes that are generated? Any programmatic means?

We have not exposed anything. Happy to chat more to understand the problem.

[gravesee]

Thank you for the suggestion. This will work for cases where tags vary little for nodes with shared logic, but say I have a dozen nodes generated from a parameterized_sources decorator - all with identical pocessing logic but different inputs and different data governance needs.

I am wondering if it is possible to apply the machinery of the tag decorator to nodes that are not yet materialized. I have not started digging into the source code to determine if this is possible - or if a custom decorator is what I should look into.

[skrawcz]

Yep understood. See @elijahbenizzy 's comment below, but we think we should be able to add:

@tag_outputs(
    public_usage={'usage' : 'public'}, 
    private_usage = {'usage' : 'private'})
@parameterize_sources(
    **{"public_usage": {"invar": "outvar1"}, "private_usage": {"invar": "outvar2"}}
)
def outvar_logic(invar: pd.Series) -> pd.Series:
    # logic goes here
    return invar

Where @tag_nodes takes in a mapping of created node_name: {dict of tags}. Thoughts?

[elijahbenizzy]

Idea here -- don't think this would be too hard. I'm happy to prototype it out in the next little bit. If its as easy as I think I have a sizable release coming soon and I can tack it on :) #254 (comment)

[elijahbenizzy]

Adding to this -- we're looking at ways to make querying nodes easy. If this would be helpful, feel free to leave feedback! #159

[elijahbenizzy]

So yeah, as @skrawcz said applying different tags to different nodes produced by the same function is not currently feasible. How's this for a possible change though... The nice thing is that this just generally works for any decorators that output multiple nodes...

@tag_outputs(
    public_usage={'usage' : 'public'}, 
    private_usage = {'usage' : 'private'})
@parameterize_sources(
    **{"public_usage": {"invar": "outvar1"}, "private_usage": {"invar": "outvar2"}}
)
def outvar_logic(invar: pd.Series) -> pd.Series:
    # logic goes here
    return invar

Thoughts on this API? Its a little verbose but quite flexible -- works for extract, etc...

[elijahbenizzy]

Hey @gravesee -- implemented this. Its in rc version sf-hamilton==1.12.0rc3. Would love feedback! Ended up calling the decorator tag_outputs. PR if you're curious: #256.

To install: pip install sf-hamilton==1.12.0rc3.

[gravesee]

Works great! It is verbose, but it will work well for my use case! We store metadata for our features in yaml file. Loading one into a tags dict and splatting it into this decorator works beautifully.

[elijahbenizzy]

Awesome! Will probably be doing a little Xmas day release :)

Re:yaml files the goal of Hamilton is to make it so everything is present in code -- e.g. all the metadata is next to the functions, which forms the single source of truth. So, I'd be really curious as to what metadata you can't express now that could fit into functions.

Ideally the data in the yaml fiel would be reportable/derivable from the info in the code...

[gravesee]

I think it's partly the verbosity of specifying tags for functions decorated with extract_columns. I have a dataset schema with over 1,000 columns. My data governance peers would like to add rich metadata to these columns so we can track their usage in downstream interemdiate and final variables that end up in models. This would yield a huge dictionary that takes up a lot of screen real estate especially after formatted with Black.

For the extract_columns case I'm thinking of breaking up the one function into many separate functions, all with the same body, but with smaller extract columns decorators (and therefore smaller tag outputs).

For example:

@tag_outputs(
   ...
)
@extract_columns('var1', 'var2', 'var3')
def vars_of_one_logical_grouping(df: pd.DataFrame) -> pd.DataFrame:
    return df

@tag_outputs(
   ...
)
@extract_columns('var4', 'var5', 'var6')
def vars_of_another_logical_grouping(df: pd.DataFrame) -> pd.DataFrame:
    return df

This would allow us to group things logically and put them in related modules. Is this a pattern you would recommend? Is there a performance penalty to splitting out extract_columns from the same dataset into separate functions?

[skrawcz]

I think it's partly the verbosity of specifying tags for functions decorated with extract_columns. I have a dataset schema with over 1,000 columns. My data governance peers would like to add rich metadata to these columns so we can track their usage in downstream interemdiate and final variables that end up in models. This would yield a huge dictionary that takes up a lot of screen real estate especially after formatted with Black.

Yes that's an interesting scaling issue. In my mind, it's either we make it easy to link the YAML file (or parts of it), or put it into a separate python file and import it to link with the code... 🤔 we'll defer to you here for ergonomics insights. So let us know what is/isn't working.

This would allow us to group things logically and put them in related modules. Is this a pattern you would recommend?

Yes I think modules are a good logical unit to group things into. You just need to align on the boundaries and naming with your team. Do provide feedback around how this impacts your workflow/operations. Since the goal with Hamilton is to enable teams to own code/concerns overtime; we think a little more upfront development/organizing pays dividends in the long run -- but it takes a while for feedback on the long run to get back to us 😉 .

Is there a performance penalty to splitting out extract_columns from the same dataset into separate functions?

There shouldn't be. We just wrap the output dataframe with a function that pulls out the relevant column without any copying.