All Systems Operational

About This Site

System Status for FogBugz and Kiln.

FogBugz Website   ? Operational
FogBugz API   ? Operational
Kiln Website   ? Operational
Kiln Repositories - HTTPS   ? Operational
Kiln Repositories - SSH   ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
FogBugz Website Response Time
Fetching
FogBugz API Response Time
Fetching
Kiln Website Response Time
Fetching
Past Incidents
Dec 16, 2018

No incidents reported today.

Dec 15, 2018

No incidents reported.

Dec 14, 2018
Resolved - Outage has been resolved, outage duration was 0:37.
sites are up and back to normal
Dec 14, 17:40 EST
Update - Problems identified:
Kiln has performance issues.
Actions taken:
Almost all customers are up, team is on a call to confirm
Estimated duration: 60 minutes.
Dec 14, 17:06 EST
Update - Problems identified:
Kiln has performance issues.
Actions taken:
Skype group has been created and investigation started. More details will be provided soon.
Estimated duration: 60 minutes.
Dec 14, 16:34 EST
Identified - Starting December 14, 2018 21:21 GMT we are experiencing an incident: Kiln Website performance degraded. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 60 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Dec 14, 16:24 EST
Resolved - Outage has been resolved, outage duration was 3:51.
System is up.
Dec 14, 12:16 EST
Update - Actions taken:
Team has fixed the issue for 15 customers. Fixing the remaining 26 ones.
Dec 14, 12:00 EST
Identified - Starting December 14, 2018 16:26 GMT we are experiencing an incident: FogBugz Website is unavailable. This is affecting restricted set of customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Dec 14, 11:29 EST
Resolved - Outage has been resolved, outage duration was 3:51.
All affected repositories are back online.
Dec 14, 12:14 EST
Update - Problems identified:
30 customers can't access their repositories
Actions taken:
Team has fixed the issue for 15 customers. Fixing the remaining 15 ones.
Estimated duration: 60 minutes.
Dec 14, 11:53 EST
Update - Problems identified:
30 customers can't access their repositories
Actions taken:
36 customers were impacted. Team has already fixed 2 and working on fixing the remaining ones.
Estimated duration: 30 minutes.
Dec 14, 11:09 EST
Update - Problems identified:
30 customers can't access their repositories
Actions taken:
Root cause was identified and the issue for one customer was fixed. Working on fixing remaining ones
Estimated duration: 30 minutes.
Dec 14, 10:54 EST
Update - Problems identified:
30 customers can't access their repositories
Actions taken:
Team is checking Redis database.
Estimated duration: 30 minutes.
Dec 14, 10:37 EST
Update - Problems identified:
30 customers can't access their repositories
Actions taken:
Team is checking the communication between the web servers. Investigation is undergoing.
Estimated duration: 30 minutes.
Dec 14, 10:00 EST
Update - Impact confirmed to be multiple customers. We are working on identifying the scope.
Dec 14, 09:35 EST
Update - Problems identified:
30 customers can't access their repositories
Actions taken:
This seems to be a particular issue as no other reports were received. Checking system and services.
Estimated duration: 30 minutes.
Dec 14, 09:34 EST
Update - We are continuing to work on a fix for this issue.
Dec 14, 09:14 EST
Update - Problems identified:
Kiln has performance issues
Actions taken:
Team is in a call checking what might have caused this issue.
Estimated duration: 60 minutes.
Dec 14, 09:09 EST
Identified - Starting December 14, 2018 13:20 GMT we are experiencing an incident: Kiln Website is unavailable. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 60 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Dec 14, 08:28 EST
Completed - The scheduled maintenance has been completed.
Dec 14, 06:59 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 14, 06:30 EST
Scheduled - On date December 14, 2018 11:30 GMT, FogBugz Website will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: December 14, 2018 11:30 GMT
Scheduled End Date: December 14, 2018 11:59 GMT

What is being addressed: *Summary*: Changing timeout value for fogbugz shopping website.

*Start*: 2018-12-14 11:30 AM UTC 
*End*: 2018-12-14 11:59 AM UTC 
*Target*: Fogbugz Shop
*Expected downtime*: 0 minutes
Dec 14, 06:11 EST
Dec 13, 2018
Resolved - Outage has been resolved, outage duration was 2:07.
Services are up.
Dec 13, 16:37 EST
Update - Problems identified:
The network issue is on the VMWare side.
Actions taken:
Services are up. Waiting some time for confirmation.
Estimated duration: 30 minutes.
Dec 13, 16:27 EST
Update - Problems identified:
The network issue is on the VMWare side.
Actions taken:
Team is working together with Central Virtualization team to fix it.
Estimated duration: 60 minutes.
Dec 13, 15:54 EST
Update - Problems identified:
Seems to be Network issue.
Actions taken:
Team is checking to fix network issue.
Estimated duration: 30 minutes.
Dec 13, 15:33 EST
Update - Problems identified:
Seems to be Central DB issue.
Actions taken:
Team is checking the db with Central team.
Dec 13, 15:10 EST
Identified - Starting December 13, 2018 19:27 GMT we are experiencing an incident: FogBugz Website is experiencing intermittent failures. This is affecting restricted set of customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Dec 13, 14:29 EST
Dec 12, 2018

No incidents reported.

Dec 11, 2018

No incidents reported.

Dec 10, 2018

No incidents reported.

Dec 9, 2018
Resolved - Outage has been resolved, outage duration was 4:53.
All services are back online and the issue is fixed.
Dec 9, 14:15 EST
Update - Problems identified:
An infrastructure issue is causing the incident. A critical server was accidentally shut down.
Actions taken:
Team is waiting for an update from the support .
Estimated duration: 300 minutes.
Dec 9, 13:39 EST
Update - Problems identified:
An infrastructure issue is causing the incident. A critical server was accidentally shut down.
Actions taken:
Reprovisioning infrastructure - The responsible team has been contacted to start this server.
Estimated duration: 300 minutes.
Dec 9, 12:45 EST
Update - Problems identified:
An infrastructure issue is causing the incident. A critical server has been accidentally shut down
Actions taken:
The responsible team has been contacted to start this server
Estimated duration: 30 minutes.
Dec 9, 11:57 EST
Identified - Starting December 09, 2018 12:59 GMT we are experiencing an incident: FogBugz Website is unavailable. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Dec 9, 08:01 EST
Dec 8, 2018

No incidents reported.

Dec 7, 2018

No incidents reported.

Dec 6, 2018
Resolved - Outage has been resolved, outage duration was 0:25.
All services are up
Dec 6, 05:46 EST
Identified - Starting December 06, 2018 10:27 GMT we are experiencing an incident: FogBugz Website is unavailable. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 60 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Dec 6, 05:29 EST
Dec 5, 2018

No incidents reported.

Dec 4, 2018
Resolved - Outage has been resolved, outage duration was 0:13.
SQLNY9 customers got an outage
6 customer trial database
Only 1 customer is really affected
Dec 4, 04:04 EST
Identified - Starting December 04, 2018 08:37 GMT we are experiencing an incident: FogBugz Website performance degraded. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Dec 4, 03:41 EST
Dec 3, 2018

No incidents reported.

Dec 2, 2018

No incidents reported.