Data Miner Release Notes

Production Release Date:  10.26.2023

About This Release

Enhancements to Instantaneous Load Feed

Existing Feature Improvements

Instantaneous Load feed: After this release, following updates will be made:

• Date timestamps will end on 5s and 0s
• UG zone will be added
• Feed description will be enhanced to indicate the region to zone summation
• Seven-Day Load Forecast

Data Miner 2.0 API Guide PDF

Production Release Date:  7.19.2023

About This Release

Reverting decimal precision on existing feeds indicated below to what they were prior to the EMS Upgrade.

Existing Feature Improvements

The columns in the feeds indicated below will post data with the decimal precision indicated after this release.

1. For Feed Area Control Error, the column Area Control Error will now have decimal precision of 2.
2. For Feed Operational Reserves, the column Reserve MW will now have decimal precision of 2.
3. For Feed Instantaneous Wind Generation, the column Wind Generation MW will now have decimal precision of 0.
4. For Feed Instantaneous Dispatch Rates, the column Instantaneous Dispatch Rate will now have decimal precision of 0

Training Release Date:  2.8.2023
Production Release Date:  2.16.2023

About This Release

Two New Feeds. Enhancements to two feeds.

New Features

• Hourly Marginal Emission Rates - This feed presents the Marginal Emission Rates by hour. The data is an hourly roll up of the five-minute emission rates presented in the Five-Minute Marginal Emission Rates.
• Hourly Total Emissions - This feed shows the estimated total emissions broken down by fuel type for each hour.

Existing Feature Improvements

• Generation by Fuel Type - The generation megawatts for this feed currently represent the generation for the interval closest to the top of hour. With this enhancement, the generation values are being updated to the generation over the previous 12 intervals within the hour. Please note that due to delay in IT SCED case processing, some hourly values may not be the average of full 12 intervals.
• Real-Time Unverified Five Minute LMPs - With this enhancement, date time parameter will be mandated input for the API call. This enhancement is being made to improve the performance of the data retrieval via API.

Data Miner 2.0 API Guide PDF

Release Date
12.15.2022

About This Release

Enhancements to two feeds.

Existing Feature Improvements

New column to present the subzones will be added to both Reserve Subzone Buses and Reserve Subzone Resources feeds with this release.

Data Miner 2.0 API Guide PDF

Release Date
11.17.2022

About This Release

One New feed.

New Features

Annual Zonal NSPL feed provides the network service peak load MW values by transmission zone.

Data Miner 2.0 API Guide PDF

Release Date
10.20.2022

About This Release

Enhancement to two feeds.

Existing Feature Improvements

• Real-Time Ancillary Service Market Results - The posting frequency of this feed will be updated from Monthly to daily with this release. This feed will run on business days and post data after the verification process. The column descriptions are being updated for better clarity.
• Day-Ahead Ancillary Service Market Results - With the release, the Tier 1 MW column will be removed from this feed. The column descriptions will be updated for better clarity.

Release Date
Training Release Date: 9.16.2022
Production Release Date: 9.29.2022

About This Release

New feeds and Updates to existing feeds as part of reserve Price Formation efforts.

Feed Name	Changes	Feed Short Name for API Integration
Real-Time Ancillary Service Market Results	Data Granularity is changing from hourly to five minutes New subzones will be added . Feed Long Name updated for clarity purposes	reserve_market_results
Day-Ahead Ancillary Service Market Results	This new feed will contain the results of the Day-Ahead Ancillary Service Markets	da_reserve_market_results
Day-Ahead Scheduling Reserve	This feed will be retired. No new data will be posted after September 30, 2022	None
Dispatched Reserves	New reserve types and areas will be added. Two Additional columns will be added. Existing Reserve Types Names are updated from plural to singular for consistency purposes	dispatched_reserves
Non-Synchronized Reserve	This feed will be retired. Data posted data prior to October 1, 2022. that requires updates will still be posted this feed but no data will be posted for days after October 1, 2022	None
Operational Reserves	Changes to reserve names and addition of new reserve names. Please refer to Data Miner FAQ for old reserves names to new reserve names mapping	operational_reserves
RTO Synchronized Reserve Preliminary Billing Data	This feed will be retired. Will be retired. Data posted data prior to October 1, 2022. that requires updates will still be posted this feed but no data will be posted for days after October 1, 2022	None
Real-Time Dispatched Reserves	Addition of new reserve type and area. One new column will be added. Existing Reserve Types Names are updated from plural to singular for consistency purposes	rt_dispatch_reserves
Synchronized Reserve Events	Addition of new reserve type and area	sync_reserve_events
Real-Time Ancillary Service LMPs	Addition of new ancillary services. Feed Long Name updated for clarity purposes	ancillary_services
Real -Time Ancillary Services Five Minute LMPs	Addition of new ancillary services. Feed Long Name updated for clarity purposes	ancillary_services_fivemin_hrl
Day-Ahead Ancillary Service LMPs	This new feed will provide LMPs for the Day-Ahead Ancillary services	da_ancillary_services
Settlements Verified Five Minute Ancillary LMPs	This feed will be retired. No new data will be posted after September 30, 2022	None
Settlements Verified Hourly Ancillary LMPs	This feed will be retired. No new data will be posted after September 30, 2022	None
Reserve Subzone Buses	This is a new feed.	sync_pri_reserves_buses_list
Reserve Subzone Resources	This is a new feed.	sync_pri_reserves_resources_list
Synchronized Reserve Preliminary Billing Data	This is a new feed. Please refer to Data Miner FAQ for reserve products to reserve types mapping.	sync_reserve_prelim_billing
Non-Synchronized Reserve Preliminary Billing Data	This is a new feed. Please refer to Data Miner FAQ for reserve products to reserve types mapping.	non_sync_reserve_prelim_billing
Secondary Synchronized Reserve Preliminary Billing Data	This is a new feed. Please refer to Data Miner FAQ for reserve products to reserve types mapping	secondary_nonsync_reserve_prelim_billing

Release Date
9.8.2022

About This Release

Enhancements to one existing feed and one defect fix.

Existing Feature Improvements

FTR Credit Calculator Congestion LMPs: New columns are being added to this feed as part of ARR/FTR Enhancement FPA 205 filing. After the update, the data for these two new columns will be available from 6/1/2022 onwards.

Bug Fixes and Steps to Recreate

CurrentHour and LastHour Date Presets - these date filter were not bringing back data that fell within the 56th and 59th minute of the hour previously. With this fix, these date parameters will bring back data that is effective between 00 to 59:59th minute of the hour.

Release Date
7.28.2022

About This Release

Enhancements to one existing feed.

Existing Feature Improvements

Enhancements to Instantaneous Load feed. Instantaneous Load feed data posted on Data Miner posts the State Estimator MW values today. With this release, this data is going to be updated to post Generation Control system load MW values. This change will ensure load values with match between Data Miner and Data Viewer. There will no change to feed columns or structure.

Release Date
4.28.2022

About This Release

Bug fixes for one feed. TLS Protocol update.

Existing Feature Improvements

As communicated at the Tech Change Forum, PJM will be retiring obsolete TLS protocol configurations in PJM internet-facing applications. As part of this effort, Data Miner User Interface and API will stop supporting TLS 1.0 or TLS 1.1 protocols after completion of the release.

Bug Fixes and Steps to Recreate

Energy Market Generation Offers Feed:As part of PJM Price Formation efforts, PJM implemented 10 additional BID segments on ramp rate and incremental offers. This change went live via Markets Gateway on 1/5/2021. PJM has noticed that the Energy Market Generation Offers feed on Data Miner tool was not posting the data for these additional BID segments for BID dates 1/5/2021 onwards.

To correct this issue, PJM is going to update this feed columns to include the additional 10 Bid segments and associated MWs during the April release and will be reposting the data for BID date 1/5/2021 to 11/30/2021 with the additional values. Please note that the values in this columns prior to 1/5/2021 will be blank but those values do not represent a NULL MW or MWH portions of the offers.

Release Date
3.29.2022

About This Release

One New Feed Minor Updates to Feed Names and Descriptions.

New Features

New Feed - Schedule 9 rates. This feed contains the rates that are calculated for PJM Schedule 9 (Admin and Pass-Thru) and PJM Schedule 10 (Pass-thru).

Existing Feature Improvements

1. Feed description enhancements are made for following feeds:
• Load Reconciliation Billing Determinants
• Daily Load Reconciliation Billing Determinants
• Hourly Reconciliation Billing Determinants
• Monthly Reconciliation Billing Determinants Preliminary Operating Reserves Rates
2. Feed Name Enhancements are made as follows:
• Reconciliation Billing Determinants
• Monthly will be named as Load Reconciliation Billing Determinants
• Monthly Reconciliation Billing Determinants
• Monthly Reconciliation Billing Determinants Preliminary Operating Reserves Rates will be named as Operating Reserve Rates
• Preliminary

Release Date
12.9.2021

About This Release

One New Feed. Updated Frequency for one existing feed.

New Features

Day-Ahead Ratings - This feeds contains the Day-Ahead ratings used in the Day-Ahead case for the market day.

Existing Feature Improvements

Daily Generation Capacity - Previously the data used to be updated once every month. With this enhancement, data will be updated Daily.

Release Date
9.9.2021

About This Release

Five Minute Marginal Emissions. Feed Updates - Column short name update for ITSCED, RT SCED and LPC Bias feeds.

New Features

Five-Minute Marginal Emissions Feed - This data contains the marginal emissions rates for individual load nodes on the wholesale grid. For a more detailed description of the data, how it is calculated, and what it means, please review: Marginal Emissions Primer PDF

Existing Feature Improvements

Update Column short name for IT SCED Bias, LPC Bias and RT SCED Bias Feeds

Release Date
7.6.2021

About This Release

New column to one feed Metadata Updates

New Features

Fast Start Five Minute Ancillary Services LMPs - This new feed is available in Data Miner Train environment only

Existing Feature Improvements

• Real-Time Unverified Five Minute LMPs - Added new column to indicate if LPC uses the same RTSCED for pricing in more than one 5 minute interval
• Dispatched Reserves - Column name change from ‘Market Clearing Price MW’ to ‘Market Clearing Price’
  • MegaWatt is removed from the column description
• High Frequency Feeds - All feeds that are updated with an interval of 5 minutes or lower are enhanced with update availability expected information

Release Date
12.17.2020

About This Release

Four new feeds.

New Features

1. IT SCED Bias – This feed presents the IT SCED Bias values for approved IT SCED cases. Data is updated every fifteen minutes for all the approved cases within the previous fifteen minute interval.
2. RT SCED Bias - This feed presents the RT SCED Bias values for approved RT SCED cases. Data is updated every fifteen minutes for all the approved cases within the previous fifteen minute interval.
3. LPC Bias - This feed presents the LPC Bias values for approved LPC cases. Data is updated every fifteen minutes for all the approved cases within the previous fifteen minute interval.
4. Real-Time Short-Term Marginal Value Override - This feed presents the short term marginal value for constraints.

Release Date
11.30.2020

About This Release

• Three new feeds
• One enhancement

New Features

1. Real Time Dispatch Reserves - This feed presents the dispatched reserves after the Real-Time Markets verification process.
2. Performance Assessment Interval Preliminary balancing ratio - This feed presents the data for the preliminary balance ratio for the performance assessment interval. Please note that the Preliminary Balancing Ratios are being provided at the request of Market Participants and are not used in settlements. The Final Balancing Ratios are used in settlements.

Existing Feature Improvements

Reconciliation Billing Determinants Preliminary Operating Reserves Rates - Column description for DA Operating Reserve is now updated.

Release Date
10.28.2020

About This Release

Enhancements to existing feeds.

Existing Feature Improvements

1. Real-Time Unverified Five Minute LMPs - A new column is added to this feed that will represent the Output Consistency Check status on the LMP interval.
2. Real-Time Unverified Hourly LMPs - Few columns are made filterable for this feed based on member feedback.
3. Transmission Limits - Previously this feed was not being updated for intervals where there are no contingencies for that specific interval. Based on member feedback, this feed is now enhanced to post “None” as values when there are no contingencies for the time interval.

Release Date
7.30.2020

About This Release

• One new feed
• Enhancements to one feed and Bug Fixes

New Features

New Feed

Real-Time Unverified Hourly LMPs - The new feed presents hourly aggregated Real-Time unverified locational marginal pricing (LMP) data for all bus locations, including aggregates. Hourly aggregates are for the previous hour.

Existing Feature Improvements

Hourly Net Imports by State - the following enhancements have been made to this feed to get a better alignment between the feed metadata and the data presented:

a. Updated Feed Description
b. Updated column name from “net interchange” to “net exports”
c. Updated column description
d. Added filter to State column
e. Changed Feed long name to Hourly Net Exports by State

Bug Fixes & Steps to Recreate

1. Real-Time Unverified Five Minute LMPs - Previously this feed was missing data for pnodeid (along with this fix, the missing pnode data will be added).
2. Day-Ahead Scheduling Reserve - Previously there was an unwarranted lag for clearing price and DASR credit values which resulted in Data Miner presenting these values as NULLs or ZEROs. The values were being updated with the correct number somewhat later than expected timeframe. With this fix this data lag is being addressed so users can get the data sooner than before.
3. Fixed Weighted Average Aggregate Definitions - Previously the DST data for this feed was showing as missing dates. This fix address the date gaps for DST hours.
4. Load Reconciliation Billing Determinants - Hourly and Load Reconciliation Billing Determinants - Daily Feeds: Previously the Rate for Billing Determinant - 'Reactive Services' was showing value as 0 in the daily feed. Additionally for month of June, the values of 0 were being posted in the Hourly feed incorrectly. With this fix, the Reactive Services rate will show only in the Daily feed if the rate value is non-zero. The values for June in the Hourly feed were removed.

Release Date
6.25.2020

About This Release
edatafeed feeds migration to Data Miner.

New Features

14 new feeds, 3 updated feeds to present edatafeed data in Data Miner. Please see the eDataFeed to Data Miner 2 Feeds List XLS for the complete list of feeds.

Existing Feature Improvements

• FTR Auction Bids - Long-Term Auction - previously the data posted for this feed was refreshed three times per calendar year. With this release two more refresh periods are being introduced. Auction rounds 4 and 5 are being added to the data.
• Transmission Limits - Previously the contingency equipment name was separated into three different columns and separate row per contingency equipment. Based on member feedback, this release will merge the three columns into one and use a ; to separate the contingencies. The new view will be more aligned to what members see in Data Viewer and edatafeed.

Release Date
5.28.2020

About This Release
Enhancements to one feed.
Bug fixes.

Existing Feature Improvements

Hourly Demand Bid Data – the following enhancements are made to this feed to ensure the data presented via edatafeed ‘getAllDayAheadLoad’ api is presented in Data Miner:

i. Feed posting schedule is changed from Monthly to Daily.
ii. New column called Area will be added.
iii. Data was previously posted only for PJM-RTO. The data will be posted for Western Region and Mid-Atlantic Region after this release.

Bug Fixes and Steps to Recreate

• Transmission Limits - Previously this feed was showing shadow price as a positive value. With this bug fix the shadow price will be updated to a negative value following the business rules for shadow prices. Previously contingencies with no equipment information were not showing up for a biding interval. With this release, the equipment for contingencies with no equipment information will be shown as BASE. Feed column description is being enhanced.
• Ancillary Service Market Results - PJM has identified an issue with the ancillary service market results data posted to Data Miner, specifically the “Total MW” values. The issue has been corrected and data will be reposted from 10.1.2012 onwards for this feed. Please download the latest data Updated the field description for REGDMW.
• Real Time Marginal Value - PJM has identified incorrect data issues with Real-Time marginal value feed for specific dates. PJM has corrected the issue and reposted this data for the following dates: 5.1.2020, 5.4.2020, 5.5.2020, 5.7.2020, 5.8.2020, 5.11.2020, 5.15.2020, 5.19.2020, 5.20.2020, 5.21.2020, 5.22.2020, 5.23.2020, 5.24.2020, 5.25.2020 and 5.26.2020.

Release Date
4.29.2020

About This Release
Two new feeds.

New Features
New feeds:
1. Real-Time PJM MISO Market to Market Settlement Adjustments - This feed contains Hourly Real-Time MISO-PJM market-to-market settlement data for every flowgate that had coordinated operation (FFE Exchange).
2. Day-Ahead PJM MISO Market to Market Settlement Adjustments - This feed contains Hourly Day-Ahead MISO-PJM market-to-market settlement data for every flowgate that had coordinated operation (FFE Exchange).

Release Date
3.5.2020

About This Release
Back-end configuration updates with no impact to users.

Existing Feature Improvements
Configuration updates are being made on the back-end to allow adding the eDataFeed feeds. This release has no impact on the users.

Release Date
2.26.2020

About This Release
Bug fix for Data Archiving.

Bug Fixes and Steps to Recreate
On archived data, the link to the next set of results was returning fields that were not part of the search criteria. This has now been fixed.

Release Date
1.28.2020

About This Release
New State Net Interchange feed and bug fixes.

New Features
A new feed is being released in Data Miner. This data shows the amount of net imports for each state for each hour. Positive numbers represent imports into the state; negative numbers represent power being exported from the state. This data is being made available but should still be considered as a prototype and may be altered or updated in the future.

Bug Fixes and Steps to Recreate
Bug #1 - The following feeds should not have been made part of the 366 day rule. This issue has now been rectified.
Nodal Reference Prices
Fixed Weighted Average Aggregate definitions
Pricing Nodes
Market to Market Flowgate
Bug #2 - Users were experiencing an issue where repeated queries for LMP data would result in decimal points being rounded up when the last few digits were 0s. This has now been rectified and LMP values will now always have up to 6 digits after the decimal and not be rounded up.

Data Miner

Release Date
1.7.2020

About This Release
Historical Data Retrieval for LMP Feeds and a bug fix.

New Features
In order to maintain good query performance for recent data, Data Miner is implementing Data Archiving for older historical data for the larger data feeds. This will allow the retrieval of historical data while still preserving performance for more recent, frequently queried data. Data Archiving is being implemented for the following feeds: Real-Time Five Minute LMPs, Real-Time Hourly LMPs and Day-Ahead Hourly LMPs.
The impact of Data archiving are noted below. A guide on historical data has been published on the Data Miner Tools page.
Historic data will be available via UI and API as it is today
The data structure, format and layout has not changed
Historic data start and end dates are rolling dates, not calendar dates
Historic data has fewer querying options
API request and response structure, format and field definitions have not changed
Appropriate error messages will be returned to users if UI and/or API query contains a filter that is not available for historic data

Bug Fixes and Steps to Recreate
The validation for date range to not exceed 366 was not working for feeds that had the EPT and UTC as datetimestamps (instead of just date). This has now been fixed, and users attempting to use a date range more than 366 days for any feed will receive validation errors.

Historical Data Guide PDF

Release Date
10.23.2019

About this release
One New Feed and API Bug Fix.

New Features
New Feed
The following feed is being made available as part of FERC Order 844. October 2019 data for these feeds will be presented monthly from November 20th onwards. Note that data prior to October 2019 is still considered confidential and therefore there will be no data back-loaded for this feed.
Operator Initiated Commitments - This new feed will have economic max and reason for the Operator Initiated Commitments by zone as required by FERC 844.

Bug Fixes and Steps to Recreate
Sorting on DateTime Beginning UTC - Previously the sorting functionality on DateTime Beginning UTC parameter was not working as designed. With this fix, the sorting will work as designed on UTC dates.

Release Date
7.17.2019

About this release
Nine new feeds and bug fix for wind generation feed.

New Features
New Feeds
As noted in Release 19.01 Release Notes, the following feeds are being made available as part of FERC Order 844. July 2019 data for these feeds will be presented monthly from August 10 onwards. Note that data prior to July 2019 is still considered confidential and therefore there will be no data back-loaded for these feeds.
Daily Uplift by Zone - This feed will be retired and will be replaced by new feed called Uplift Credits by Zone. The new feed will have substantively the same information but the Credit Type pivoted into one single column. The Daily Uplift by Zone feed will be moved to the new ‘Retired’ category from ‘Generation’. This change was made to be consistent with the formatting of new data that will be required by FERC Order 844. While PJM tries to avoid substantially changing the structure of existing feeds, it was felt that in the long term, improved consistency in the uplift-related feeds would be of benefit to users.
Uplift Charges by Zone - This new feed will have zonal uplift charge data as required by FERC Order 844.
Generator Specific Uplift Credits - This new feed will have generator specific uplift credits on a monthly basis as required by FERC Order 844 and is posted on a one-month lag. Data for July 2019 will not appear until the September posting.
Load Response Uplift Credits - This new feed will have demand response specific uplift credits on a monthly basis as required by FERC Order 844 and is posted on a one-month lag. Data for July 2019 will not appear until the September posting.
The following feeds will present the data that’s currently available on the Operational data page. Data layout changes have been made to this data to make the layout easier to understand and maintain, as well as reducing the amount of code changes needed for members in the event that changes are made in the future. Various sections of the Operational Data page have been split into 5 different feeds. Data for these feeds will be available from July 17 onwards.
Unverified Five Minute LMPs (Category: Locational Marginal Prices) - This feed contains the unverified real-time LMP data for a subset of pricing nodes which are zonal, aggregate, interface, hub and 500 Kv bus.
Transfer Interface Information (Category: System Information) - This feed contains the actual flow MW and transfer limits information for the Transfer Interfaces.
Instantaneous Load (Category: Load) - This feed contains the load MW for four regions. Loads are calculated from raw telemetry data and are approximate. The displayed values are NOT official PJM Loads.
Transmission Limits (Category: System Information) - This feed contains the current PJM transmission limits. This feed will be populated with data only when there are transmission constraints for that five minute interval.
Instantaneous Dispatch Rates (Category: System Information) - This feed contains the PJM instantaneous dispatch rates. The following dispatch rate information includes both the PJM electronic dispatch signals and manual dispatch signals. The electronic rates are updated every 15 seconds. The manual rates are updated with a frequency of 1 to 5 minutes and reflect individual generator dispatch. This data is provided for informational purposes ONLY and should not be relied upon by any party for the actual billing values. Missing interval data will not be backfilled due to the nature of the data. This data will be retained for 15 days only.
Data Layout Transformation Details
Datetime UTC and EPT stamps have been added all the feeds 500 KV Bus Locational Marginal Prices are posted under Unverified Five Minute LMP data with type = 500 KV
Current PJM Transmission Limits data is presented in a tabular form rather than a text string
Data will be retained for 30 calendar days except dispatch rates, which will be retained for 15 days
Any data for missing periods will be populated at pre-determined frequency for all the feeds except the Instantaneous Dispatch Rates. Missing data will not be populated for this feed because of the data update frequency.
New Category for the Feeds
New category called ‘Uplift’ will be added to the left navigation pane. This enhancement is to provide better grouping of uplift-related data. The above listed feeds that contain Uplift charges and credits data will be present in ‘Uplift’ category.

Fixes
Wind Generation - Previously the value “Other” in region was not being presented for this feed. With this release, the value “Other” will be present in region when appropriate.

Release Date
7.1.2019

New Features
New Feeds
Solar Generation -This feed provides the hourly solar generation amounts in PJM. The Other" area includes external units and other units where the original regional location is not known.
FTR Congestion LMPs -This feed contains the congestion locational marginal price for the FTR pricing nodes

Enhancements
Frequently Accessed Data – Previously when users clicked on any feed from Frequently Accessed Data area, the data used to open in new window. With this enhancement, the data will open in a new tab keeping user experience consistent with the Explore Data Set button.

Release Date
4.17.2019

Enhancements

Release Date
3.14.2019

About This Release
A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway screens or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release Date
1.29.2019

About This Release
A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway UI or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release Date
12.21.2018

About This Release
A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway UI or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release Date
11.30.2018

About This Release
A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway screens or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release Date
9.24.2018

About This Release
A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway UI or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release Date
8.2.2018

Fixes
Operations Summary Feeds – Previously the peak statistics data showed discrepancies between the csv file posted on the PJM website and Data Miner. The issues originate from upstream data source discrepancies. This fix will ensure the load forecast, scheduled tie flow and internal scheduled capacity will match between both data sources.
Operations Summary - Projected RTO Statistics at Peak - Enhancements are made to improve “Scheduled Tie Flow Total” field description.
Operations Summary - Projected Scheduled Tie Flow – Enhancements are made to improve “Scheduled Tie Flow” field description.
Fixed Weighted Average Aggregate Definitions – Previously the data showed discrepancies between the csv file posted on the PJM website and Data Miner. Upon further investigation, the issue was identified to be a missed posting update on the PJM website No code changes are made to Data Miner at this time.

Release Date
7.2.2018

New Features
New feeds – These feeds will be available from 7/2 onwards
Day-Ahead Temperature Sets
First Available – This is a new field that will be available for all feeds from 7/2 onwards on the feed definition page. This field will show the earliest date data is available for that specific feed.

Enhancements

Real-Time Scheduled Interchange – Enhancements made to the user interface for this feed to add a top level filter on the “Datetime Beginning EPT” column
Historical Load Forecasts – Enhancements made to the user interface for this feed to add a top level filter on the “Datetime Beginning EPT” column
Operations Summary - Projected Area Statistics at Peak – Enhancements are made to improve Data Fields descriptions
Operations Summary - Projected RTO Statistics at Peak – Enhancements are made to improve Data Fields descriptions

Known Issues

This Data Miner known issues section provides information on issues that impact users and are known to Data Miner team.

Operations Summary Feeds - There is a known issue with the peak statistics data discrepancies between the csv file posted on the PJM website and Data Miner. The issues originate from upstream data source discrepancies and are actively being investigated. Update Posted on 7/20 – The issue with forecast data is identified as the sum of area forecasts are not matching the RTO forecast. This issue stems from individual area vs aggregated zones. PJM’s load forecasting system creates forecasts for a number of different geographic areas. It creates forecasts for individual zones as well as for larger aggregated areas (such as RTO and Mid Atlantic). Due to the nature of the neural net forecasting, these larger areas do not precisely match the sum of the individual component zones. Both forecasts are correct in that they are what was forecasted by the software, but we recognize the confusion caused by the difference in the values. To make the numbers more intuitive, PJM has decided to sum up the smaller areas to get the larger area forecast, rather than providing the neural net forecast for the larger territory for the Data Miner posting. Internal scheduled capacity - The data posted in Data Miner is not matching the data posted on the PJM website. Internal scheduled capacity - The data posted in Data Miner is not matching the data posted on . The known issue with Scheduled Tie Flow data on DM2 not matching the PJM website is being investigated at this time.

Fixed Weighted Average Aggregate Definitions - There is a known issue with the termination logic for the aggregate pricing nodes that is causing additional aggregate nodes to appear. This issue is actively being worked on.

Release Date
5.31.2018

Fixes
Real-Time Scheduled Interchange Posting – Previously data for this posting was missing data for the last four hours of the day, and that data was not made available until the next day’s posting. This fix will ensure the data will be posted for all 24 hours every day.
Real-Time Five Minute LMPs – Previously, using datetime filters 20:00 to 23:00 was not returning any results. This fix will ensure this datetime filters will return data for the appropriate time block.

Enhancements
RSS Notifications – The RSS notifications for the following feeds will now include the market day the LMP data is posted for in the email body: a) Day-Ahead Hourly LMPs b) Real-Time Hourly LMPs c) Ancillary Service LMPs d) Real-Time Five Minute LMPs e) Ancillary Services Five Minute LMPs

Monthly Financial Transmission Rights Zonal LMPs – The pricing node filter for this feed will be enhanced to a text box providing enhanced search capability on the user interface.

Off-Cost Operations – Enhancements made to the user interface for this feed to add a top level filter on the “Date” column.

Release Date
4.30.2018

Fixes
Day Ahead Marginal Value and Constraints Posting - Enhancements have been implemented to reduce the time required to post the Day-Ahead Marginal Value and Constraints postings.
Market to Market Flowgate FFE – Enhancements have been implemented to move the posting time to 0830 EPT from 0900 EPT.
Settlements Verified Hourly LMPs – Updated the “Update Availability” description to “Monthly on the 4th business day at 10:00 a.m.” on the feed definition page. Note that this is a metadata update only that better reflects the actual posting time.
PJM Regulation Zone Preliminary Billing Data – The feed description has been updated to rename two columns - “Total PJM Regulation Market Capacity Clearing Price” and “Total PJM Regulation Market Performance Clearing Price” to “Total PJM Regulation Market Capacity Clearing Price Credits” and “Total PJM Regulation Market Performance Clearing Price Credits”.

Release Date
3.29.2018

New Features
New feeds – These feeds will be available from 4/2 onwards:
Real-Time Five Minute LMPs
Ancillary Services Five Minute LMPs
Settlements Verified Five Minute LMPs
Settlements Verified Five Minute Ancillary LMPs
Daily Energy Market Generation Offers

Update to existing feeds – These updates to the existing feeds will be available from 4/2 onwards:
Real Time Marginal Value - As part of five minute settlements initiative, the shadow prices will be posted at a five minute granular level. Historic data will remain hourly.
Real Time Transmission Constraints – This feed will be retired as the data is now included in the Real Time Marginal Value feed. No new data will be posted to this feed. Historic data will remain available.
PJM Regulation Zone Preliminary Billing Data – As part of five minute settlements initiative, two new columns (Total PJM Regulation Market Capacity Clearing Price and Total PJM Regulation Market Performance Clearing Price) were added to the feed. Historic data for these new columns will be BLANK.
RTO Synchronized Reserve Preliminary Billing Data – As part of five minute settlements initiative, one new column (Total Subzone Tier 2 PJM-Scheduled MWh) are added to the feed. Historic data for this new column will be BLANK.
Day-Ahead LMP Posting: The posting of the Day-Ahead LMPs will now synchronize with other PJM Tools

Real-Time LMPs Posting: Enhancements are done to reduce the time required to post RealTime LMPs

Performance enhancements to five minute LMP data downloads - As part of the five minute settlements initiative, PJM has enhanced the performance of Data Miner. PJM has performed load testing of Data Miner. When 50 users simultaneously download the full prior day of five minute LMPs for all pricing nodes, it takes an average of 4.5 minutes to complete. This test case represents the expected typical scenario. The same test with a more extreme case of 250 simultaneous users yields download times of approximately 12 minutes. Naturally these times will vary based on system usage, the speed of your internet connection and other factors. These times are not a guarantee, but should help users set their expectations. Data Miner uses caching to more quickly provide the most recent day of data; users querying older historical data should expect it to take longer.

PJM.com Website Enhancement - The left navigation area where users select the feeds was previously not sorted. With this enhancement, the feeds within each category will be alphabetically sorted.

Fixes
Operations Summary Feed schedule - Previously the operations summary feeds were running at 8 AM to post only the 8 AM data. This scheduler is being modified to post Operations Summary data every hour starting from 5 AM to 8 AM. The 8 AM data will only be retained. This matches the functionality for the Operations Summary on the PJM website.
Date Alias “This Month” - Previously the date alias “ThisMonth” returned results that did not include the last four hours of the month. This fix will ensure “thismonth” will return results for the entire datetime within the calendar month.
Pricing node Selector not working in Internet Explorer Browser - Previously pricing node selector was not returning any results when users tried to select pricing nodes on LMP feeds using Internet Explorer. This fix will ensure the pricing node selector will work as desired.
Inconsistency in the zone assigned to pricing nodes in Data Miner's LMP output - Previously some pricing nodes in the LMP historic data were showing zones as AES instead of AEP. This fix will correct the zones for the relevant pricing nodes.

Release Date
3.29.2018

New Features
New feeds – These feeds will be available from 4/2 onwards Real-Time Five-Minute LMPs Ancillary Services Five-Minute LMPs Settlements Verified Five Minute LMPs Settlements Verified Five-Minute Ancillary LMPs Daily Energy Market Generation Offers Production: April 24, 2019
Update to existing feeds – These updates to the existing feeds will be available from 4/2 onwards
Real Time Marginal Value - As part of five minute settlements initiative, the shadow prices will be posted at a five minute granular level. Historic data will remain hourly. Real Time Transmission Constraints – This feed will be retired as the data is now included in the Real Time Marginal Value feed. No new data will be posted to this feed. Historic data will remain available. PJM Regulation Zone Preliminary Billing Data – As part of five minute settlements initiative, two new columns (Total PJM Regulation Market Capacity Clearing Price and Total PJM Regulation Market Performance Clearing Price) were added to the feed. Historic data for these new columns will be BLANK RTO Synchronized Reserve Preliminary Billing Data – As part of five minute settlements initiative, one new column (Total Subzone Tier 2 PJM-Scheduled MWh) are added to the feed. Historic data for this new column will be BLANK
Day-Ahead LMP Posting: The posting of the Day-Ahead LMPs will now synchronize with other PJM Tools
Real-Time LMPs Posting : Enhancements are done to reduce the time required to post RealTime LMPs
Performance enhancements to five minute LMP data downloads As part of the five minute settlements initiative, PJM has enhanced the performance of Data Miner. PJM has performed load testing of Data Miner. When 50 users simultaneously download the full prior day of five minute LMPs for all pricing nodes, it takes an average of 4.5 minutes to complete. This test case represents the expected typical scenario. The same test with a more extreme case of 250 simultaneous users yields download times of approximately 12 minutes. Naturally these times will vary based on system usage, the speed of your internet connection and other factors. These times are not a guarantee, but should help users set their expectations. Data Miner uses caching to more quickly provide the most recent day of data; users querying older historical data should expect it to take longer.
UI Enhancement The left navigation area where users select the feeds was previously not sorted. With this enhancement, the feeds within each category will be alphabetically sorted.

Fixes:
Operations Summary Feed schedule Previously the operations summary feeds were running at 8 AM to post only the 8 AM data. This scheduler is being modified to post Operations Summary data every hour starting from 5 AM to 8 AM. The 8 AM data will only be retained. This matches the functionality for the Operations Summary on the PJM website. Date Alias “ThisMonth” Previously the date alias “ThisMonth” returned results that did not include the last four hours of the month. This fix will ensure “thismonth” will return results for the entire datetime within the calendar month. Pricing node Selector not working in Internet Explorer Browser Previously pnode selector was not returning any results when users tried to select pricing nodes on LMP feeds using Internet Explorer. This fix will ensure the pricing node selector will work as desired. Inconsistency in the zone assigned to pricing nodes in Data Miner's LMP output Previously some pricing nodes in the LMP historic data were showing zones as AES instead of AEP. This fix will correct the zones for the relevant pricing nodes A validation in Markets Gateway has been preventing some resources from updating their energy parameters (economic min/max, emergency min/max) in real time because of comparisons to the SR Max value, which is an hourly value locked 65 min before the operating hour. With this new release, this validation will no longer occur in the Markets Gateway UI or via XML. Instead, the validation will be performed downstream in the optimization engines.

New Features
The validation in Markets Gateway for the following rules: 1) SR Max = Eco Max ≤ Emergency Max for all resources, and 2) SR Max ≤ Eco Max ≤ Emergency Max for resources that have submitted a modification request, has been removed. If a parameter is submitted on any Markets Gateway screen that violates these rules, Markets Gateway will accept the submission but the optimization engines will use the correct value. Participants are encouraged to submit and update their parameters in a timely manner to reflect the actual capability of their resources.
New/modified XML: No changes are required for XML submission

Associated External Interface Specification Guide version: Revision 61

Release Date
2.15.2018

Fixes
API Portal Documentation Update Previously the attribute descriptions on the API management portal had generic descriptions. Based on user feedback, in this release, these descriptions have been enhanced and will be published on the API portal. This same information will also be published in the API Guide available on the PJM website.

Release Date
2.1.2018

Pricing node termination date logic
Previously the pricing nodes that should have one long, continuous active period instead have several contiguous periods. This issue was being caused by how the pricing node feed was being updated based on LMP data for that pricing node. This fix will ensure that all active pricing nodes will have a continuous active period. Existing data will remain unchanged however this fix will take effect for all data going forward
Daily Generation Capacity missing data
A small number of missing intervals for the months of December 2017 and January 2018 were identified for the Daily Generation Capacity Feed. These data gaps were corrected.
Date Alias “Last Week” is incorrect [API Only]
Previously the Date Alias “Last Week” was not bringing back a full 7 days’ worth of data. This fix ensures that this date alias works as intended. The date alias “Last Week” will now return data where the date falls within the prior calendar week. This fix is implemented for all date aliases

Release Date
1.11.2018

New Features
Feed Update Notifications: This enhancement provides users with a RSS Notification they may subscribe to. Once subscribed, the RSS feed will provide notifications whenever a feed is updated. Users will need to subscribe to each feed separately. This feature has been added based on member feedback.
Metadata for Real-Time and Day-Ahead LMP Reposts: This enhancement provides users with details on LMP reposts, similar to what is available on the PJM website today. The details will contain the specific dates and the pricing node ids LMPs were reposted for. Notices will be posted every time there is a LMP repost by PJM. On the screen, this information will be displayed on the feed definition page. On the API, it will be available via the feed_metadata API response.

Fixes
Datetime Filters on UI and API Previously, the hour 00:00 was being considered as hour 23:00 for the Datetime Start and End filters on EPT and UTC dates. When users entered the hour 00:00 as the End date, 24 hours of data is being returned instead of just the hour 00:00. The EPT hour 19:00 was also being read as EPT hour 23:00 because of a defect in EPT to UTC conversion. This fix will resolve these issues and few other usability issues with the datetime filters. Please refer to the following discussion topic posted on Tech Change Community page for specifics on the datetime filters behavior after the release: https://pjm.force.com/techchangeforum/s/question/0D50V00003ghJkrSAE/
Reset Button on the screen previously, clicking on Reset Button on the screen completely erases the datetime filters. This fix will ensure that the reset button will now bring back the default filters that appear upon page load.
Duplicate records in some feeds appear sporadically There is a processing bug that is casing duplicate records to appear in specific feeds. Enhanced logging has been added to provide further diagnostic information to investigate this issue so that it may be addressed in the future.