Removing null rows

CloverDX Platform
1

I am new to clover and cannot figure out how to do what I want. I am trying to build a data warehouse by first building the dimension tables. I am reading in the fact data for one column (community) and then comparing that to the community dimension table. I have used a DataIntersection and found all rows in the fact data for community id that do not exist in the community dimension table. Then I use Reformat and Dedup to get the community ids that I need to insert into the community dimension table. But some of the rows of my fact data have nulls for community id and I want to exclude those from consideration. I was thinking I could do it in the Reformat, or maybe in a ExtFilter. In the Reformat step I currently strip off all the columns from the fact metadata to just the community id column. Then I want to do something like:
if $in.0.community_id is null
then remove that row
else include the row

Then when I get to the Dedup step, it will remove all the duplicates and I will have all the new community ids to insert into my community dimension table. I am just stuck on the syntax to use.
Thanks!

2

Hi, slaurenc,

I would use just ExtFilter component with a condition like: !isnull($in.0.community_id)

Records with community_id not equal to null will be sent to the first output port (output port 0). You can also connect some edge to the optional second output port (output port 1) and records with community_id equal to null will be sent there.

Best regards,

3

Thank you! I’m not much of a programmer and I’m finding it difficult to see examples of everything I want to do. Once I’ve seen an example I can extrapolate it out, but figuring out the syntax is a challenge.
Serena