Big with all 0 wav clips in calculateAverageSpectra causing errors
Oops changing fix so that only non-noise clips with all 0 get dropped
addRecordingsupdateFiles paths in some edge casesfuture package frameworkAdding runIciReview function and better functionality for
runDepthReview
Finally adding default option to PAMpalSettings and addFunction to
just use all defaults (mostly for my convenience when testing and the
values dont matter)
calculateEchoDepthmode='fixed' option to processPgDetections that lets users create
fixed time length events, processing all data similar to mode='recording'
but organizing into uniform time chunksplotDataExplorer to use shiny instead of manipulate since
the manipulate version stopped working a long time agoaddWaveHeight,
calculateEchoDepth, filterEchoDepths, and runDepthReviewDropping stringr dependency
Adding CSV option for addHydrophoneDepth
Bugfix for addNotes
Added fillZeroes option to getClipData
plotGram works with gaps in data
Adding cmap options to plotGram and calculateAverageSpectra
Adding various image control options to plotGram
Updated filter to work more consistently with complicated queries
Added check for processing mode='time' where it would previously
crash if $db was a folder, not a file
calculateAverageSpectra was not properly scaling the power spectrum, giving
values ~50 dB higher than it shouldexport_annomate bugged file writing\(c) usage in matchTimeDataaddSettingsaddRecordingsProbably about time this was 1.0.0, just in time for a paper!
Updating front page readme
NA valuesaddMeasures and getMeasures functionstimeJoin from matchTimeData so other packages can usegrouping='clickTrain' when processing
with mode='db'. Events have prefix CT instead of OE or DGL. By default mode='db' will
process all three of grouping=c('event', 'detGroup', 'clickTrain')sampleDetector function to subsample by detectoraddFPOD and getFPODData functions to deal with adding/getting FPOD
data from AcousticStudiesUpdating getClipData and writeEventClips to have a fixLength option that
allows users to create clips of a fixed length using just the buffer argument
Allowing processPgDetections to take multiple label arguments to specify
different table columns if using both DG and OE events in same db
Updating markAnnotated to give warnings on NA values instead of failing
Updating roccaWhistleCalcs to avoid divide by zero NA/Inf values
markAnnotated function to flag whether or not detections are within annotation
boxesAdding matchTimeData function to add arbitrary data to AcousticStudies
Bug fixes for some random stuff
Updated to warnings in addGps to now be less overwhelming
Fixed bug for displaying some notes after combining with bindStudies
addGps now adds gpsUncertainty which says how many meters since to
the closest GPS point. Useful to catch points where we have interpolated a lot.
Adding addAnnotation and related functions for annomate exporting
Explicitly marking joins that are expected to have multiple matches re:
dplyr::join's new behavior
Smol change to not grab unnecessary columns in addGps
Big speed improvements for processing with mode='time' and mode='recording'
Converting all paths to use '/' instead of '\'
Found out updateFiles was unuseably slow, so I fixed it. I dunno, maybe 1000x faster
depending on how many files you have?
Removing the $source item in the settings slot for each AcousticEvent. These were not used for anything, and when the source was wav files it would end up with a huge repetitive list for every event. Ended up taking up a huge amount of memory, removing this results in AcousticStudys that are half the size.
Updated filter function for a 2x speed increase. Turns out quos(...) is expensive when you need to do it thousands of times, so now passing along the character version of dots when we can instead of recomputing it.
JK pdate again filter is actually 10x faster. Way better to pull out all the data into a single df and then just reassign everything to the proper event
Updated localizations slot to use more standardized names for imported PGTargetMotion locs. Currently using latitude, longitude, perpDist, perpDistErr, depth, depthErr. These seem like good standard localization columns going forward. Changed the ppVars()$tarMoCols to pull in only columns corresponding to these, and added new ppVars()$locCols to reflect new standard column names
Added addNote, getNotes, removeNote functions for adding and viewing notes. Can be added to study or event, and will print when they are printed.
addDatabase not working properly for directory of dbsaddGps when using a dataframe as the source, thresh argument was not doing anythingmyGram function used for power spectrum calculations to normalize by average window energy instead of average window amplitude because we are almost always using power spectraaddHydrophoneDepth to be able to take numeric input for constant depthChanging added column fro depth to hpDepth in addHydrophoneDepth
writeEventClips now has a filter option to apply low/high/bandpass filters
getSr and matchSR functions were not working properly in some edge cases
changed getDetectorData to return db for more usefulness
getBinaryData returns UIDs in the order given
mode='db' to not repeat UIDs within a single eventMissed a bug in new addRecordings
addBinaries now works with vector of folders
Changing addRecordings to return NA values when reads are not possible instead of omitting
Some fixes to plotGram - previously whistle contours could plot at wrong frequnecy and time scale could be thrown off
addDatabase can now accept a folder containing databases
cepstrum calculations handled better internally
Detection Group Localiser had changed in PAMGuard - reworked code to work with new naming scheme and fail less often if some tables are missing
filter has more regular behavior - no longer returns NULL if no detections are present, was previously causing odd behavior in loops
Event order is now preserved when doing mode='time'
Added functions nDetections, nClicks, nWhistles, nCepstrum, nGPL that get total number of detections in an event or study
getICI for type='data' returns one big dataframe instead of lists of lists of dataframes
Added showBreaks option to calculateAverageSpectra to toggle showing line breaks for multiple events in concatenated spectrogram
Added bindStudies function to combine multiple AcousticStudy objects
Trying to stabilize processing against minor errors - should no longer crash entire processing if one piece of binary data fails, will try again instead
Trying to fix some crashes with missing Sound Acquisition data
Bug with mode='time' where sample rate could be read in as a character, causing crashes
calculateAverageSpectra would have problems when doing multiple events that had detections with repeated UIDs
matchEnvData now has a depth parameter to specify depth or range of depths to match
Store and print total processing time for processPgDetections
PAMpalSettings callAdded support for GPL Detector data!
Added processDate to ancillary info of each study
Removing some unnecessary package dependencies
testClick from 192kHz to 500kHz to avoid unnecessary errors when trying to add filters > 96kHzbrightness and contrast parameters to calculateAverageSpectra. These only affect the concatenated spectra plot. Also added a ylim parameter for the mean spectra plot.calculateAverageSpectra, previously was not working with event names as an argument for evNumcalculateAverageSpectra was always computing noise spectrum cuz I'm bad at stuff
Bug fix for addRecordings with certain SoundAcquisition table setups
getBinaryData was improperly complaining about multiple matches
checkStudy complains about too many things, shush alreadyNormalizing some paths internally
checkStudy checks for NA values and reports
Small bug in addRecordings that wasn't assigning startSample properly for separate wav files
Trying to re-work how we can consistently access SR after processing. Functions should now be able to automatically tell if a decimated sample rate is appropriate (calculateAverageSpectra and writeEventClips mainly) for clicks in binaries as long as XML settings have been added with addSettings before processing.
PAMpalSettings has new parameter settings to add XML settings without an additional call to addSettings
filter works better with detector names
Grouping in mode='recording' wasn't working well when recordings were processed with "Merge contiguous files" checked, fixed now
writeEventClips works with decimated data now with useSample = TRUE
Added more info to loadPamguardXML. Now loads in click sweep classifier type data as well as pre-filter and trigger filter data
standardClickCalcs no longer fails if upper end of filter >= to Nyquist
processPgDetections with mode='time' can run successfully if no Sound_Acquisition table is present by providing columns db and sr the grouping file (db not needed if processing a single database)
processPgDetections with mode='db' can run successfully if no Sound_Acquisition table is present, will ask for sample rate during processing
Missed some function warnings with new warning system
calculateAverageSpectra now draws lines breaking up events in concatenated spectrogram if plotting multiple events at once
plotGram now uses start and end times based on the start of the event, not the recording
standardClickCalcs incorrectly had outputs labelled centerHz_3dB and centerHz_10dB when units were actually in kilohertz. Names have been changed to centerkHz_3dB and centerkHz_10dB
calculateAverageSpectra has a few new features. plot can be a vector of lenght two specifying which of the two plots to create, useful when only one is desired. New parameters mode and decimate added. mode allows users to specify 'spec'(default) or 'ceps' to calculate either spectrum or cepstrum of signal. decimate reduces the samplerate of the signal by an integer factor before calculations, this can be especially useful for cepstrum data.
Fixed bug in processPgDetetions for detection group localiser data crashing with an error related to tarMoCols, and also updated code to automatically check for event labels. Will look for (in order) Text_Annotation column, column with species in the name, column with label in the name, or column with id in the name. If none of these exist, defaults to the first non-standard column present in the database.
filter has new special case to filter by detector names
processPgDetections rare bug fix with processing bad event grouping files for mode='time' and mode='recording'
getBinaryData can now select for detection type, other functions updated to use this
filter works better with species, in particular can do is.na(species) now
addHydrophoneDepth function added that will try to read depth data from a database
processPgDetections for mode='db' now adds target motion localizations if they are present and stores in the localizations slot of events
addRecordings stores some sample number and samplerate data
writeEventClips has new option useSample to try and use the startSample values stored in binary files instead of just UTC times
addRecordings tries to use to get start sample data from database
Fixed bug where processPgDetections would crash if no comment field was present
calculateICI error fixed when exactly two detections were present for a detector
export_banter reports names of event level measures it finds
Switching from using eventUID to event ID for all labeling and id() purposes
Added ability to work with Pamguard's XML settings files with functions addSettings, removeSettings, and loadPamguardXML. Currently this lets PAMpal tell if a decimator was in use and adjust the sample rate accordingly, no longer need to manually set the sr_hz parameter for standardClickCalcs if XML settings have been added to the pps.
PAMpalSettings objects now have a @settings slot to accommodate this addition, older objects will need to have this slot manually added or they will give warnings of Not a validObject(), no slot of name "settings" for this object of class "PAMpalsettings"
Added updatePamObject function to deal with updating S4 classes created in older versions of PAMpal
Added message on load to warn about updating objects
Added new warning system that captures warnings during processing and function calls and stores them in the ancillary slot of the AcousticStudy object. These can be accessed with the new getWarnings function
calculateICI keeps UID info with the datacalculateAverageSpectra has a sort option to sort clicks by peak frequency in the concatenated spectrogram
processPgDetections with mode='time' does a better job of parsing sample rates from database
Added mode='recording' to processPgDetections that will group events by the start and end times of wav files used
processPgDetections will try to auotmatically determine an appropriate mode if none specified
processPgDetections with mode='db' now reads in the comment column of the database and stores in the ancillary() slot of each event
addRecordings reads in an extra format of dates [0-9]{14}_[0-9]{3}
Channels for clicks are read in separately from standardClickCalcs, now these will be saved even if no calc functions are present along with UID and UTC
checkStudy behaves better if run on a study with no events
calculateAverageSpectra doesn't fail on high SNR values anymore
plotWaveform and friends have a length argument and will now attempt to get sample rate values from the AcousticStudy object if possible
getBinaryData tries to get sample rate better
plotGram added in preliminary form. Plots spectrograms and cepstrograms of events, can also overlay WMD detections onto plot.
calculateAverageSpectra - more improvements. Noise shouldnt fail anymore, added SNR threshold argument, returns all noise data and UIDs. Output names changed from all and average to allSpec and avgSpec.calculateAverageSpectra has a channel argument, better color scaling for concatenated spectrogram, and a beta version of average noise floor
processPgDetections will now automatically use mode='time' if a grouping argument is provided and is either a dataframe or a valid filepath
processPgDetections now properly updates progress bar for mode='time' when some binary files have no data or are not within any events, previously would not get to 100% or would jump
calculateAverageSpectra now averages in exponential space rather than in dB space, and plots are created in separate plotting windows. The norm option now just scales to a maximum dB value of 0Multiple functions that tried to match files against a list would fail if filepaths contained "\" as file separators, these have been changed to use fixed=TRUE so that this no longer happens
Can now set function parameters for functions being adding during the initial call to PAMpalSettings and addFunction instead of being forced to go through interactive menus
Cepstrum module can now be named "BurstPulse" or "Burst Pulse" in addition to "Cepstrum" to be recognized by PAMpal
Function checker was set up poorly and would unintentially write to global env on some failed checks
calculateAverageSpectra was not passing along manual sr input properly
processPgDetections with mode='time' handles date converting better, will no longer fail with Excel rounding 12:10:00 to 12:10. Also removed some unnecessary interactive steps, will now stop and ask user to provide instead of asking user to provide during call
Fixed bug in calculateICI if large values were present rounding would lead to bad answers. Now filters results by z-score < 2 before taking ICI to get rid of outlier large values
Fixed bug in writeEventClips that was undoing buffer in the wrong direction if the buffered time was not present in any of your wav files, would previously get a false warning of "cant find wav file"
Updated how warnings work for writeEventClips, now always reported even on failure
Adding filter options to calculateAverageSpectra similar to standardClickCalcs
Fixed bug where processing with mode='time' was leaving in full binary path names instead of just file basename, messing up other stuff like getBinaryData. Fixed and temporary adjustment to getBinaryData made
getBinaryData was not doing a good job if binaries had the same name. It do better now
processPgDetections will only store file names that it actually used instead of all possible matching file names, related to other binary fix.
Added getICI, getClickData, getWhistleData, and getCepstrumData as easier accessors
Updating webpage with calculateICI and getDetectorData guides
writeEventClips now attaches clip times with _YYYYMMDDHHMMSS_mmm to the end of file nameswriteWignerData warns about bad directories instead of crashing
filter fixed for bug in species and database filtering if these keywords were present on the right hand side of the call
writeWignerData functionnewUIDs not getting used properly for some users because they were integer64
fixed bug with writeEventClips not reporting all file names if multiple databases present
addRecordings prints some failure warnings
filter for databases also removes those databasess from the files(x)$db
writeEventClips updated to use channel argument
R and package version now stored in studies when created
standardClickCalcs now uses "channelMap" from binaries to assign correct channel if it is present instead of defaulting to Channel 1 & 2
export_banter warning messagaes cleaned up
Removed extra "Peak 3dB" and "Peak 10dB" values from standardClickCalcs
filter now works with a special case for database and also works for any environmental variables added
Updating some docs for GitHub pages website
Adding calculateAverageSpectra back
Modified processPgDetections for mode='db' to work with a possible "newUID" column from PAMmisc's updateUID function. Also does a better job of matching if there are multiple possible binary files.
addGps now asks if you want to add gps data to your database if it has none
Better warning messages for processing data
writeEventClips modified to work with events or individual detections
addRecordings function added to deal with keeping track of wav recordings in a study
Name changed to PAMpal because pamr was taken on CRAN
Lots of prep for CRAN submission
Changes to standardClickCalcs so it doesnt crash if peak frequency is 0, happens if lower filter bound is 0
Added updateFiles function to update file locations of a study object if you have changed computers or moved folders around
Added checkStudy function to do some sanity checking for possible issues after data is processed. Currently only checks for peak freq of 0 which would mean you probably want a different filter value or add a calibration function
Added test datasets to inst/extdata to go along with added testthat support
NEW FUN FUNCTION calculateAverageSpectra! Calcultes and plots average spectra from an event! Woohoo!
Fixed issue with export_banter not finding ancillary measures properly
matchEnvData compatible with update in PAMmisc package, works with custom functions now
setSpecies handles things more gracefully without provided method argument
Changed setSpecies to choose insted of stop when multiple methods provided (default)
Added filter method that works like dplyr's filter for detections in an object. Has a special case when specifying species or Species that will filter an AcousticStudy object by the species listed in the $id spot of each species slot
Also added first version of addAnnotation function, it asks a lot of questions for now
AcousticStudy [ accessor changed to return another AcousticStudy after subsetting the events in that study
writeEventClipsFixed bug in addGps that could create duplicate rows after adding gps to your data. Would occur if two GPS entries occurred within the same second, now adds milliseconds before matching so that this is avoided.
Added support for SoundTrap files for writeEventClips with format='soundtrap'
Added better progress bar for mode='db' that is based on binary files not databases
Added writeEventClips function for creating wav clips of events. Currently needs some adjustments but should work for Pamguard data if your selection of wav files definitely covers all of your events
Fixed bug where click detectors were being improperly named if a matched classifier was present in the binary file
matchEnvData extends a function of the same name in the "PAMmisc" package (as of PAMmisc v1.4.1). This lets you download environmental data and match it to your AcousticStudy and AcousticEvent objectsexport_banter if an event had exactly 1 detection fixedexport_banter for list of AcousticStudy objects not propagated event names properlyexport_banter can take list of AcousticStudy objects now, and fixed a bug where NA values were being introducedSkip over binaries with no UIDs in mode='db'
mode='db' was not properly passing along an ID if you supply it
export_banter now lists which measures were NA in the $na dataframe output
egClicksError message in processPgDetections for mode='db' now reports db and binary file failed on better
Changed folder choosing funciton for addBinaries, previous version was windows-only and would not properly start in current working directory
addDatabase now prints added databases for reference
setSpecies now has method = 'reassign' that can reassign species id to new ones provided in a dataframeprocessPgDetctions bug fixed when encountering a binary with 0 detectionsprocessPgDetections shouldnt crash on empty binary file that also didnt finish writing the header and footer for mode='time'processPgDetections handles binary files that never finished writing without a mysterious crash now. Should only affect mode='time'.standardClickCalcs changed to have a filterfrom_khz and filterto_khz option so that a bandpass filter can be applied. If only a highpass filter is desired simply leave filterto_khz as the default NULLaddCalibration reworked slightly again, created calibration function now takes frequency in Hz as input and outputs dB correction to add instead of taking in result of seewave::spec and outputting corrected spec. This makes it more flexible and means we aren't calculating spectrum twice for no reason
standardClickCalcs adjusted to use new calibration, and temporarily has a new parameter calculated - dBPP - which is the peak to peak value as traditionally defined in acoustics. This is mostly for testing comparability to other Matlab based code of new calibration stuff. Also the spectrum is never rescaled to a max of 0 like it was previously, hopefully should represent accurate SPL levels with new calibration code
addCalibration now asks for units of your calibration, and should properly add or subtract this value from calculated spectrum depending on units selectedexport_banter now has an option to split your data into training and test sets by specifying a proportion for the training argument, and now puts out a pretty table showing a summary of your events and speciesprocessPgDetections has some minor updates to improve clarity when binary files or databases aren't found using mode='time'getPgDetections has now been re-named to processPgDetections to more accurately reflect what it is doing. Functions starting with get___ will be used just for accessing data
showWaveform and related functions have been renamed to plotWaveform
General naming consistency overhaul - sr and db should now be used in place of sampleRate and database wherever these were previously used, and this should be the naming convention going forward. This means included functions will need to be re-added (they previously relied on it being named sampleRate), and any custom functions should expect data to contain the item sr and not sampleRate
AcousticStudy class has been added. This will now be the class of object returned by processPgDetections, it stores your list of AcousticEvent objects with other important data.
Any data processed in previous versions will need to be re-run, and all functions in older PAMrSettings files will need to be removed and re-added
New Function: calculateICI calculates the inter-click interval of click data, and stores it in the ancillary slot of an AcousticEvent.
New Function: plotDataExplorer creates an interactive plot to allow users to explore the data in an AcousticStudy, AcousticEvent, or data.frame. Plot can be colored or facetted by any columns that are characters or factors, and any numeric columns can be plotted with ggplot's geom_density
New Function: getDetectorData gathers all detector data into single data frames for each detector type ('click', 'whistle', and 'cepstrum') for easier manipulation
Detector dataframes in AcousticEvents now have a 'calltype' associated with them, currently one of 'whistle', 'click', or 'cepstrum'. This allows for future fun stuff to happen
getBinaryData will now attempt to get the appropriate sample rate for each data point, either from the settings or matching by time using the database file if more than one sample rate was in your data.
Many speed improvements - click calculations should take about half the time, and processPgDetections with mode='time' will now skip over binaries that are outside of the time range of specified events
getPgDetections no longer has an option mode='all', and it is no longer necessary to supply a sample rate for mode='time'. It will now attempt to match events to a correct database used on time stamps, the match the appropriate sample rate the same way that mode='db' does. If a manual sample rate needs to be applied it should now be entered into the grouping table
Functions addBinaries and addDatabase can now add files from another PAMrSettings object, and will report how many files have been added when finishing
export_banter will now attempt to find event level measures in the 'measures' item in the ancillary slot of each AcousticEvent.
export_banter no longer has a reportNA option, any NA values that are removed are now always saved to a separate 'na' item in the list output. The function also reports how many detections were processed.
addGps now also stores all of the gps data loaded into the gps slot of the AcousticStudy instead of only adding coordinates to detections
getPgDetections will name events with database appended for method = 'db' instead of just event ID number to ensure uniqueness across multiple databasessetSpecies can now use a dataframe for method = 'manual', and has a top secret option for SRexport_banter with options to exclude certain species and to export data without species codes to use for prediction only instead of training a banter modelBetter error tracking when functions cause getPgDetections to crash
Now reads in angles and angleErrors from click data
export_banter allows you to specify certain columns to not export
'time' mode for getPgDetections will now report a sample event time so you can see if times are being converted properly from your csv before proceeding with calculations
Fixed an issue where getPgDetections would not work if both Detection Group Localizer and Offline Click events were present in a database.
Renamed eventType and Text_Annotation columns from event databases to eventLabel within detection dataframes so there is consistency between the two
Removed some unnecessary columns from detection dataframes, including detectorName, sampleRate, Id, parentUID, and comment
Fixed a bug in how getBinaryData was checking for multiple matches on the same UID
'db'Added an id slot to AcousticEvent objects. Note that this will cause existing AcousticEvent objects to behave poorly until they have their id slot created / updated using the new setIdSlot function.
Added setIdSlot function to update older AcousticEvent objects to the new format
Changed event naming for mode='time' in getPgDetections. Will now only append numbers if event is not unique, and will also insert an underscore before the number
export_banter now properly checks for cases when there are no detectors in an event instead of crashing confusingly
export_banter no longer using list names to index and create unique event namesexport_banter now removes NA rows, and has reportNA option to see which ones are NA
Detector names have spaces replaced with _ to avoid weird issues later
Dealing with zero detection events better for export_banter
Temporary fix for click calculations with lots of zeroes in wave form - no more NA
standardClickCalcs now supports manual input of sample rate. Default argument is 'auto', which will read from the database. User can supply a numeric value for sample rate in hertz instead.Added a check in addDatabase to see if all files are actually .sqlite3 databases
Added a tryCatch in getPgDetections for mode db so that it shouldn't stop completely when encountering an error and lose all previously analysed DBs
seewave::spec, it can produce NA values for the frequency if the input wave is long. Adjusted parts standardClickCalcs and addCalibration to work around this.getPgDetections changed to work by specifying a mode as an argument instead of calling separate functions. Can now create events using a csv or dataframe with start and end times specified.
Changed export_banter to export a list with named item detectors instead of detections, no other functional change.
Added setSpecies functions for assigning species classifications to AcousticEvents.
Changed getDbData to default to looking for both OfflineEvents tables and DetectionGroupLocaliser tables, will still only load one if a specific type is provided for grouping.
Added addCalibration, applyCalibration, and findCalibration functions, as well as a plot method that will show the calibration function used. See the Calibration section above for more details.
Fixed a bug in removeFunction that would cause the incorrect number of functions to show in certain cases, and that would cause all functions to be removed when only one was selected.
standardClickCalcs has been adjusted to work with the new calibration methods. See Calibration section above for more details.
Added addGps function for adding matching GPS data to your detections. This allows you to supply a dataframe of Lat/Long locations with timestamps to match to your detections.
Added showWaveform, showSpectrogram, and showWigner functions that allow you to easily plot the waveform, spectrogram, or wigner plot of a detection in an AcousticEvent object by selecting the UID(s) you want to investigate further. getBinaryData is also added as a helper function for these, lets you easily get the binary file data for a single detection.
standardClickCalcs now supports supplying a Wave class object as input
Rocca (roccaWhistleCalcs) and cepstrum (standardCepstrumCalcs) functions added. These are also added by default to a new PRS.
changed AcousticEvent class slot name from specClass to species