{"id":5463,"date":"2017-08-06T15:01:27","date_gmt":"2017-08-06T20:01:27","guid":{"rendered":"http:\/\/www.thejuliagroup.com\/blog\/?p=5463"},"modified":"2017-08-05T15:02:15","modified_gmt":"2017-08-05T20:02:15","slug":"fixing-data-part-1-of-a-zillion-duplicate-dates","status":"publish","type":"post","link":"https:\/\/www.thejuliagroup.com\/blog\/fixing-data-part-1-of-a-zillion-duplicate-dates\/","title":{"rendered":"Fixing Data: Part 1 of a zillion &#8211; duplicate dates"},"content":{"rendered":"<p>I read a comment on line saying SAS probably would not disappear as an option for statistical analysis because &#8220;it&#8217;s good when you need to do a lot of data manipulation&#8221;.<\/p>\n<p>I wonder what world those people live in that data comes all cleanly packaged and whether they have unicorns there.<\/p>\n<p>Back on Planet Earth, I have a data set that has multiple records for the same date for the same students. \u00a0For some reason, the data were being sent at the end of each screen at one site, instead of at the end of the test. So, the data look like this:<\/p>\n<p>kat123 4 5 18 11 \u00a0 2017-04-23 17:39:26<\/p>\n<p>kat123 4 5 18 11 \u00a0 42 17 8 0 1 2017-04-23 17:41:12<\/p>\n<p>and so on.<\/p>\n<p>The students also took a post-test, months later, so &#8230;<\/p>\n<h2>I need the last record for each date, but my data has date and time<\/h2>\n<p>You might think doing<\/p>\n<p>testday= datepart(date_entered);<\/p>\n<p>would work and it would except for the fact that<\/p>\n<h2>My date is saved as a character format! What do I do?<\/h2>\n<p>You can read some suggestions here in SAS communities<\/p>\n<p><a href=\"https:\/\/communities.sas.com\/t5\/Base-SAS-Programming\/how-to-convert-char-var-to-sas-date\/td-p\/45067\">https:\/\/communities.sas.com\/t5\/Base-SAS-Programming\/how-to-convert-char-var-to-sas-date\/td-p\/45067<\/a><\/p>\n<p>I could not find<\/p>\n<p>2016-02-03 19:41:26<\/p>\n<p>and I spent a good hour trying different methods to get this to work. I will spare you the details and maybe I could have gotten some method to work (no, whatever you are considering, I probably already tried). However, this occurred to me &#8230;<\/p>\n<h2>Do you really need to change it to a date format?<\/h2>\n<p>In this case, I was not doing any calculations with the date value, I simply needed the day part as a unique value.<\/p>\n<p>I could just use the first 10 characters like this<\/p>\n<p>day_of_test = substr(date_entered,1,10) ;<\/p>\n<p>If you figured this out in the first sentence or two you are probably laughing by now (shut up). \u00a0Yes, it doesn&#8217;t matter if it is formatted as a date or not.\u00a0So, that is what I did. \u00a0After creating a variable that is just the day of the test, I sorted by username, day of test and date entered (which included the time value). Then, I read in the data using the BY statement in the Data step so there would be \u00a0last. variable created that is whether or not this is the last record with that value in the BY group. \u00a0I output the last record for each day by using a subsetting IF statement.<\/p>\n<p>Data fixdata ;<br \/>\nset mydata.aztech_pre ;<\/p>\n<p>*** CREATE day_of_test variable as characters 1-10 \u00a0;<br \/>\nday_of_test = substr(date_entered,1,10) ;<\/p>\n<p>*** SORT by username, day of test and date entered (including time);<br \/>\nproc sort data=fixdata;<br \/>\nby username day_of_test date_entered ;<\/p>\n<p>*** DATA step that only saves last record ;<br \/>\nData mydata.aztech_pre ;<br \/>\nset fixdata ;<\/p>\n<p>*** \u00a0BY statement to define that the data is by username and day_of_test ;<br \/>\n*** NOTE: \u00a0If you didn&#8217;t do the PROC sort first, this won&#8217;t work. For shame! ;<br \/>\nby username day_of_test ;<\/p>\n<p>***<br \/>\nif last.day_of_test \u00a0;<br \/>\nrun;<\/p>\n<p>So, that worked perfectly. I included my missteps because it is easy when you are a newbie to believe that everyone is smarter than you and never makes bonehead mistakes. Not so. We all make them all of the time. The important thing is, figuring it out in the end. Sometimes the easy way is not so obvious.<\/p>\n<p>Or, maybe it is and I&#8217;m a bonehead. Either way, it worked. Now on to step 2.<\/p>\n<p>&nbsp;<\/p>\n<p><a href=\"http:\/\/www.7generationgames.com\/buy\/\">When I am not writing about SAS, I&#8217;m making games that teach math, social studies and language.<\/a><\/p>\n<p><a href=\"http:\/\/www.7generationgames.com\/buy\/\">Check them out.<\/a><\/p>\n<p><a href=\"http:\/\/www.thejuliagroup.com\/blog\/wp-content\/uploads\/2017\/06\/Screen-Shot-2016-05-03-at-2.44.33-PM.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-5428\" src=\"http:\/\/www.thejuliagroup.com\/blog\/wp-content\/uploads\/2017\/06\/Screen-Shot-2016-05-03-at-2.44.33-PM.png\" alt=\"screen shots from our games\" width=\"658\" height=\"275\" srcset=\"https:\/\/www.thejuliagroup.com\/blog\/wp-content\/uploads\/2017\/06\/Screen-Shot-2016-05-03-at-2.44.33-PM.png 658w, https:\/\/www.thejuliagroup.com\/blog\/wp-content\/uploads\/2017\/06\/Screen-Shot-2016-05-03-at-2.44.33-PM-300x125.png 300w\" sizes=\"auto, (max-width: 658px) 100vw, 658px\" \/><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I read a comment on line saying SAS probably would not disappear as an option for statistical analysis because &#8220;it&#8217;s good when you need to do a lot of data manipulation&#8221;. I wonder what world those people live in that data comes all cleanly packaged and whether they have unicorns there. Back on Planet Earth,&#8230;<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-5463","post","type-post","status-publish","format-standard","hentry","category-dr-de-mars-general-life-ramblings"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/posts\/5463","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/comments?post=5463"}],"version-history":[{"count":3,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/posts\/5463\/revisions"}],"predecessor-version":[{"id":5466,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/posts\/5463\/revisions\/5466"}],"wp:attachment":[{"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/media?parent=5463"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/categories?post=5463"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.thejuliagroup.com\/blog\/wp-json\/wp\/v2\/tags?post=5463"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}