FogBugz and Kiln On Demand Status

Dec 16, 2018

No incidents reported today.

Dec 15, 2018

No incidents reported.

Dec 14, 2018

Kiln Website performance degraded - Kiln has performance issues - KILN-1513

Resolved - Outage has been resolved, outage duration was 0:37.
sites are up and back to normal
Dec 14, 17:40 EST

Update - Problems identified:
Kiln has performance issues.
Actions taken:
Almost all customers are up, team is on a call to confirm
Estimated duration: 60 minutes.
Dec 14, 17:06 EST

Update - Problems identified:
Kiln has performance issues.
Actions taken:
Skype group has been created and investigation started. More details will be provided soon.
Estimated duration: 60 minutes.
Dec 14, 16:34 EST

Identified - Starting December 14, 2018 21:21 GMT we are experiencing an incident: Kiln Website performance degraded. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 60 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Dec 14, 16:24 EST

FogBugz Website is unavailable - Kiln On-Demand Performance Issue - MANUSCRIPT-14741

Resolved - Outage has been resolved, outage duration was 3:51.
System is up.
Dec 14, 12:16 EST

Update - Actions taken:
Team has fixed the issue for 15 customers. Fixing the remaining 26 ones.
Dec 14, 12:00 EST

Identified - Starting December 14, 2018 16:26 GMT we are experiencing an incident: FogBugz Website is unavailable. This is affecting restricted set of customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Dec 14, 11:29 EST

Kiln Website is unavailable for 30 customers KILN-1507

Resolved - Outage has been resolved, outage duration was 3:51.
All affected repositories are back online.
Dec 14, 12:14 EST

Update - Problems identified:
30 customers can't access their repositories
Actions taken:
Team has fixed the issue for 15 customers. Fixing the remaining 15 ones.
Estimated duration: 60 minutes.
Dec 14, 11:53 EST

Update - Problems identified:
30 customers can't access their repositories
Actions taken:
36 customers were impacted. Team has already fixed 2 and working on fixing the remaining ones.
Estimated duration: 30 minutes.
Dec 14, 11:09 EST

Update - Problems identified:
30 customers can't access their repositories
Actions taken:
Root cause was identified and the issue for one customer was fixed. Working on fixing remaining ones
Estimated duration: 30 minutes.
Dec 14, 10:54 EST

Update - Problems identified:
30 customers can't access their repositories
Actions taken:
Team is checking Redis database.
Estimated duration: 30 minutes.
Dec 14, 10:37 EST

Update - Problems identified:
30 customers can't access their repositories
Actions taken:
Team is checking the communication between the web servers. Investigation is undergoing.
Estimated duration: 30 minutes.
Dec 14, 10:00 EST

Update - Impact confirmed to be multiple customers. We are working on identifying the scope.
Dec 14, 09:35 EST

Update - Problems identified:
30 customers can't access their repositories
Actions taken:
This seems to be a particular issue as no other reports were received. Checking system and services.
Estimated duration: 30 minutes.
Dec 14, 09:34 EST

Update - We are continuing to work on a fix for this issue.
Dec 14, 09:14 EST

Update - Problems identified:
Kiln has performance issues
Actions taken:
Team is in a call checking what might have caused this issue.
Estimated duration: 60 minutes.
Dec 14, 09:09 EST

Identified - Starting December 14, 2018 13:20 GMT we are experiencing an incident: Kiln Website is unavailable. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 60 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Dec 14, 08:28 EST

FogBugz Website Maintenance Window - MANUSCRIPT-14723

Completed - The scheduled maintenance has been completed.
Dec 14, 06:59 EST

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 14, 06:30 EST

Scheduled - On date December 14, 2018 11:30 GMT, FogBugz Website will undergo scheduled maintenance. The maintenance window is as follows:
Scheduled Start Date: December 14, 2018 11:30 GMT
Scheduled End Date: December 14, 2018 11:59 GMT

What is being addressed: *Summary*: Changing timeout value for fogbugz shopping website.

*Start*: 2018-12-14 11:30 AM UTC
*End*: 2018-12-14 11:59 AM UTC
*Target*: Fogbugz Shop
*Expected downtime*: 0 minutes
Dec 14, 06:11 EST

Dec 13, 2018

FogBugz Website is experiencing intermittent failures - Test site is down - MANUSCRIPT-14716

Resolved - Outage has been resolved, outage duration was 2:07.
Services are up.
Dec 13, 16:37 EST

Update - Problems identified:
The network issue is on the VMWare side.
Actions taken:
Services are up. Waiting some time for confirmation.
Estimated duration: 30 minutes.
Dec 13, 16:27 EST

Update - Problems identified:
The network issue is on the VMWare side.
Actions taken:
Team is working together with Central Virtualization team to fix it.
Estimated duration: 60 minutes.
Dec 13, 15:54 EST

Update - Problems identified:
Seems to be Network issue.
Actions taken:
Team is checking to fix network issue.
Estimated duration: 30 minutes.
Dec 13, 15:33 EST

Update - Problems identified:
Seems to be Central DB issue.
Actions taken:
Team is checking the db with Central team.
Dec 13, 15:10 EST

Identified - Starting December 13, 2018 19:27 GMT we are experiencing an incident: FogBugz Website is experiencing intermittent failures. This is affecting restricted set of customers.
We are working on identifying the root cause and fixing it.

We will provide an update every 30m or as soon as incident is resolved.
Dec 13, 14:29 EST

Dec 12, 2018

No incidents reported.

Dec 11, 2018

No incidents reported.

Dec 10, 2018

No incidents reported.

Dec 9, 2018

FogBugz Website is unavailable - Manuscript not available, showing error message: '503: Service Unavailable' - MANUSCRIPT-14571

Resolved - Outage has been resolved, outage duration was 4:53.
All services are back online and the issue is fixed.
Dec 9, 14:15 EST

Update - Problems identified:
An infrastructure issue is causing the incident. A critical server was accidentally shut down.
Actions taken:
Team is waiting for an update from the support .
Estimated duration: 300 minutes.
Dec 9, 13:39 EST

Update - Problems identified:
An infrastructure issue is causing the incident. A critical server was accidentally shut down.
Actions taken:
Reprovisioning infrastructure - The responsible team has been contacted to start this server.
Estimated duration: 300 minutes.
Dec 9, 12:45 EST

Update - Problems identified:
An infrastructure issue is causing the incident. A critical server has been accidentally shut down
Actions taken:
The responsible team has been contacted to start this server
Estimated duration: 30 minutes.
Dec 9, 11:57 EST

Identified - Starting December 09, 2018 12:59 GMT we are experiencing an incident: FogBugz Website is unavailable. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Dec 9, 08:01 EST

Dec 8, 2018

No incidents reported.

Dec 7, 2018

No incidents reported.

Dec 6, 2018

FogBugz Website is unavailable - Application is not accessible - MANUSCRIPT-14493

Resolved - Outage has been resolved, outage duration was 0:25.
All services are up
Dec 6, 05:46 EST

Identified - Starting December 06, 2018 10:27 GMT we are experiencing an incident: FogBugz Website is unavailable. This is affecting restricted set of customers.
We are working on the resolution and expect to have it resolved within the next 60 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Dec 6, 05:29 EST

Dec 5, 2018

No incidents reported.

Dec 4, 2018

FogBugz Website performance degraded - few services throws error message - MANUSCRIPT-14410

Resolved - Outage has been resolved, outage duration was 0:13.
SQLNY9 customers got an outage
6 customer trial database
Only 1 customer is really affected
Dec 4, 04:04 EST

Identified - Starting December 04, 2018 08:37 GMT we are experiencing an incident: FogBugz Website performance degraded. This is affecting all customers.
We are working on the resolution and expect to have it resolved within the next 30 minutes.

We will provide an update every 30m or as soon as incident is resolved.
Dec 4, 03:41 EST

Dec 3, 2018

No incidents reported.

Dec 2, 2018

No incidents reported.

System Status: FogBugz and Kiln

About This Site