evandertoorn
evandertoorn
Mmarimo
Created by dmad on 6/26/2024 in #help-support
Advanced memory management
now this looks absolutely delightful
10 replies
Mmarimo
Created by dmad on 6/26/2024 in #help-support
Advanced memory management
w.r.t. strict mode, copying large dataframes (i.e. 60% of RAM) would not be feasible between cells.
10 replies
Mmarimo
Created by dmad on 6/26/2024 in #help-support
Advanced memory management
The DAG could infer for further cells that the variable is no longer usable.
10 replies
Mmarimo
Created by dmad on 6/26/2024 in #help-support
Advanced memory management
Example use case:
# import cell
import polars as pl
import seaborn as sns
# raw parsing cell
huge_df = pl.read_parquet("huge.parquet")
# plot 1.
sns.histplot(huge_df, ...)
# plot 2.
sns.boxplot(huge_df, ...)
# For further exploration, we actually only need a subset
partial_df = huge_df.filter(...)
partial_df2 = huge_df.group_by(...).agg(...)
# Potentially, I'd be fine with indicating that this is where the need for it to exist stops
mo.drop(huge_df)
# ... further analysis
# import cell
import polars as pl
import seaborn as sns
# raw parsing cell
huge_df = pl.read_parquet("huge.parquet")
# plot 1.
sns.histplot(huge_df, ...)
# plot 2.
sns.boxplot(huge_df, ...)
# For further exploration, we actually only need a subset
partial_df = huge_df.filter(...)
partial_df2 = huge_df.group_by(...).agg(...)
# Potentially, I'd be fine with indicating that this is where the need for it to exist stops
mo.drop(huge_df)
# ... further analysis
10 replies