Anyone have experience with the read_patterns() function in SafeGraphR?

Help
#1

Anyone have experience with the read_patterns() function in SafeGraphR? I’m getting the following error “Error in data.table::setkeyv(patternsb, by) : some columns are not in the data.table: customer_USER_ID” from

IMB_Patterns_1 <- read_many_patterns(filelist,
                                     dir = 'C:/Users/seancrotty/ReadManyInputs/Test/',
                                     recursive = TRUE,
                                     select = variables_1,
                                     multi = list(
                                       list(name = 'patterns_by_store',
                                            by = 'customer_USER_ID'),
                                       list(name = 'by_CBG',
                                            by = 'poi_cbg')
                                     )



                                     )
#2

The column names included in your multi argument must be correct. Check the Patterns docs for the proper column names. Also we don’t have by-customer data.

#3

The data is from the safegraph match service, the outputs from which include columns from the customer’s input dataset. That’s where the key matches are coming from, but I also ran the code with just the placekey as “by” column, as well as by = poi_cbg and got the same error. The columns are definitely part of the data in every .gz.csv file, but perhaps the package presets the acceptable column headings? My normal troubleshooting process on stack overflow etc didn’t turn up anything helpful, so I appreciate any ideas.

#4

Oh I see. Hmm, this function predates the match service and I’ve never tested it with match service data. I should test it with that. It may be that the function just doesn’t work with match service data yet

#5

If it is just a problem with the match service data, I can find a work-around for my project, but it’d be good to know either way.

#6

I’m planning to check with a match service example soon. For now, I’d recommend something like this (may need some tweaking):

library(data.table)
library(SafeGraphR)
library(purrr)

flist <- list.files(info for locating the files here)

reader <- function(x) {
    print(x)
    d <- fread(x, select = c('customer_USER_ID','poi_cbg','by_CBG','patterns_by_store'))
    d1 <- expand_integer_json(d, 'patterns_by_store', by = 'customer_USER_ID')
    d2 <- expand_integer_json(d, 'by_CBG', by = 'poi_cbg')
    return(list(d1, d2))
}

dat <- flist %>%
    map(reader) %>%
    rbind_by_list_pos()
#7

Hmm, I just got some example data from the match service and it looks like regular Shop data. I still don’t see the columns you’re referring to but have you tried read_many_shop instead of read_many_patterns? It’s designed for shop-format data.

#8

Hi Nick, I’m happy to share a copy of input and output data with you if it will help determine if the package can work with the match data as it exported in my case (which I can’t figure why would be different than another match-data use-case). I’ll try read_many_shop as well and see if that works.

#9

Sure, send it along