gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6022 commits 28 branches 278 tags 135 MiB
Commit graph

370 commits

Author SHA1 Message Date
LazyDBA247-Anyvision 7f458d5e78 better postgres connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of stale connections.
 2021-02-15 07:45:09 +02:00
LazyDBA247-Anyvision 51b4963e2e postgres2 & memsql2 
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
 2021-02-14 13:14:36 +02:00
Chris Lu 7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Konstantin Lebedev 94eac4f00e Do reconnect to RabbitMQ 
https://github.com/google/go-cloud/issues/2958
https://github.com/chrislusf/seaweedfs/issues/1773
 2021-02-11 13:59:36 +05:00
Chris Lu 821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu c3af72d950 fier store: fix elastic search regression 
fix https://github.com/chrislusf/seaweedfs/issues/1774
 2021-02-04 01:30:14 -08:00
Chris Lu 2396ac234c filer: fix elastic search pagination 
possible fix for https://github.com/chrislusf/seaweedfs/issues/1774
 2021-02-02 11:34:12 -08:00
Chris Lu 4be51c0701 filer: leveldb and hbase may miss files when listing large directories more than 1024 
fix https://github.com/chrislusf/seaweedfs/issues/1768
 2021-01-31 20:11:44 -08:00
Chris Lu 314dc1c957 filer: etcd store fix listing 
fix https://github.com/chrislusf/seaweedfs/issues/1767
 2021-01-30 13:38:44 -08:00
Chris Lu 990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu 00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu 6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu 096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu d5add83e85 filer store: add postgres2 2021-01-19 18:07:29 -08:00
Chris Lu 52a8f1470e filer store: add mysql2 2021-01-19 17:21:50 -08:00
Chris Lu 93b3adba98 fix bucket creation 2021-01-19 15:55:51 -08:00
Chris Lu fa0c8d5283 fix error 2021-01-19 14:05:48 -08:00
Chris Lu 4c5b752b04 restructuring sql stores 2021-01-19 13:53:16 -08:00
Chris Lu ca8f793978 refactor: prepare for bucket specific tables 2021-01-19 12:34:58 -08:00
Chris Lu 90ce1eec19 add more input params 2021-01-19 11:32:55 -08:00
Chris Lu de876c795d minor fix 2021-01-18 01:14:27 -08:00
Chris Lu 09f49d1c04 refactoring 2021-01-16 19:52:15 -08:00
Chris Lu a4063a5437 add stream list directory entries 2021-01-15 23:56:24 -08:00
Chris Lu 01dc8a43ba fix rocksdb 2021-01-15 18:31:29 -08:00
Chris Lu 16ad74f477 go fmt 2021-01-14 23:11:27 -08:00
Chris Lu f002e668de change limit to int64 in case of overflow 2021-01-14 23:10:37 -08:00
Chris Lu 19e52fd414 pass along hasMore 2021-01-14 22:49:29 -08:00
Chris Lu c0bcf6f6e1 implement sql changes 2021-01-14 22:44:22 -08:00
Chris Lu 5d4568b91f implement elastic changes 2021-01-14 22:42:25 -08:00
Chris Lu 893cbc8482 implement c* changes 2021-01-14 22:38:34 -08:00
Chris Lu 5ef43b9b09 implement etcd changes 2021-01-14 22:35:56 -08:00
Chris Lu 54527f0326 implement hbase changes 2021-01-14 22:34:35 -08:00
Chris Lu b5ceffe188 implement leveldb changes 2021-01-14 22:33:05 -08:00
Chris Lu c64bfb0e2e implement mongodb changes 2021-01-14 22:28:51 -08:00
Chris Lu 34a846009d implement rocksdb changes 2021-01-14 22:24:26 -08:00
Chris Lu d2ece29fff implement redis changes 2021-01-14 22:21:31 -08:00
Chris Lu 9a50dbcda0 chagned api 2021-01-14 22:18:38 -08:00
Chris Lu e1c7bc66f9 change interface 2021-01-14 22:17:49 -08:00
Chris Lu 4aaa97c789 skip not found error in case of race condition 2021-01-14 19:56:14 -08:00
Chris Lu fd1d8a2a09 fix compilation 2021-01-14 00:06:15 -08:00
LazyDBA247-Anyvision 8eed763b97 better mysql connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of connections.
 2021-01-14 08:14:21 +02:00
Chris Lu f17aa1d06c skip directory loop if dropping the bucket 2021-01-13 13:49:04 -08:00
Chris Lu ca73013453 leveldb3: ensure repeatable adding removing buckets 2021-01-13 13:48:48 -08:00
Chris Lu 5cd9c9e642 remove println 2021-01-12 18:47:38 -08:00
Chris Lu def6d8e565 fix test 2021-01-12 12:32:54 -08:00
Chris Lu 629c9962e7 filer: add leveldb3 
support dedicated leveldb instance for each bucket
 2021-01-12 02:29:44 -08:00
Chris Lu 361043e6c1 filer store: leveldb2 fix nil entry error if not found 2021-01-12 02:28:57 -08:00
Chris Lu cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu acc0d5cd2c Revert "Revert "filer store: redis module upgrade"" 
This reverts commit dd0171e35c.
 2021-01-11 02:30:19 -08:00
Chris Lu dd0171e35c Revert "filer store: redis module upgrade" 
This reverts commit 40161c9cec.
 2021-01-11 02:09:18 -08:00
Chris Lu 394513f598 filer: ensure seamless meta data updates 2021-01-11 02:08:55 -08:00
Chris Lu e2e0e30a51 filer: redis default to false for useReadOnly and routeByLatency 2021-01-08 01:12:44 -08:00
Chris Lu 40161c9cec filer store: redis module upgrade 2021-01-08 01:08:44 -08:00
Chris Lu 07f4703bfc
Merge pull request #1738 from qieqieplus/rocksdb 
fix rocksdb crash when list directory
 2021-01-07 02:26:41 -08:00
qieqieplus 0f18592315 alter style 2021-01-07 18:21:48 +08:00
qieqieplus c7e7431fd3 fix seek lastkey may reach EOF 2021-01-07 17:07:56 +08:00
Chris Lu e327385644 go fmt 2021-01-06 04:22:00 -08:00
Chris Lu 2b76854641 add "weed filer.cat" to read files directly from volume servers 2021-01-06 04:22:00 -08:00
qieqieplus 0764fccde7 ignore decode error for non-entry data 2021-01-06 17:42:43 +08:00
qieqieplus b2e50f602f
fix time comparison 2021-01-05 19:36:40 +08:00
qieqieplus bbae13e415 impl: TTL per entry for rocksdb; fix package name 2021-01-05 17:58:40 +08:00
qieqieplus ebb223c190 fix for rocksdb kv 2021-01-04 21:21:46 +08:00
chenqieqie 40b3207f18 fix #1726 2021-01-04 18:48:55 +08:00
Chris Lu bdd166c031 adjust readme 2021-01-03 00:47:58 -08:00
Chris Lu f4d87f8b9d works 
performance seems about the same as leveldb for insert entries
 2021-01-02 23:32:58 -08:00
Chris Lu 2c3c2c27d7 separate prefix from namePattern 
fix https://github.com/chrislusf/seaweedfs/issues/1722
 2021-01-01 20:23:23 -08:00
Chris Lu da7e5aaa65 filer: optimize for less number of directory lookup 
bottom up directory lookup
 2020-12-27 20:46:17 -08:00
Chris Lu da87f6b265 remove unused code 2020-12-26 15:21:12 -08:00
Chris Lu 0a7c5f85a9 filer: add namePattern to search in current folder 2020-12-26 15:05:31 -08:00
Chris Lu 0e016bc7bd hbase add ttl 2020-12-24 12:10:35 -08:00
Chris Lu 9bf6c10505 fix prefix 2020-12-24 00:18:59 -08:00
Chris Lu 2fea8cfa0f fix compilation 2020-12-23 23:54:15 -08:00
Chris Lu 53bc1ea25b fix compilation 2020-12-23 23:53:46 -08:00
Chris Lu b5e2be635a adjust for directory listing 2020-12-23 23:49:22 -08:00
Chris Lu c4a202ec41 fix wrong column family 2020-12-23 23:23:05 -08:00
Chris Lu 94e3757c08 fix 2020-12-23 21:45:16 -08:00
Chris Lu 64c48c9724 seems compiling 2020-12-23 21:44:53 -08:00
Chris Lu 1c7e1295dc Merge branch 'master' into support_ssd_volume 2020-12-22 17:44:52 -08:00
Chris Lu a0990b929d minor 2020-12-22 17:33:40 -08:00
Chris Lu 97e3432dfe avoid wrong error 
fix https://github.com/chrislusf/seaweedfs/issues/1691
 2020-12-22 02:46:24 -08:00
Chris Lu 4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu 6c4f32d173 remove unused code 2020-12-22 02:33:31 -08:00
Chris Lu 90d785a15f filer: redis, redis cluster, cassandra support super large directory 2020-12-22 02:26:05 -08:00
Chris Lu fe46411cd4 handle directory listing edge cases 2020-12-21 23:37:43 -08:00
Chris Lu c74bede730 handle listing directories 2020-12-21 23:19:05 -08:00
Chris Lu adf8cb4000 filer: path-specific stores trim out common prefixes 2020-12-21 22:57:13 -08:00
Chris Lu 0823bde8d8 refactor: split into two files 2020-12-21 22:05:15 -08:00
Chris Lu 8ab70ccf76 refactor 2020-12-21 21:46:32 -08:00
Chris Lu 65cc6dc636 refactor to dedicated function 2020-12-21 21:18:34 -08:00
Chris Lu 488c2680e8 change default filer store root 2020-12-21 21:15:22 -08:00
Chris Lu f5a19a87be filer: report error if the path specific store is not found 
related https://github.com/chrislusf/seaweedfs/issues/1691
 2020-12-21 12:43:52 -08:00
Chris Lu 41c0f3ad24 filer: support path-specific filer store 2020-12-19 01:27:09 -08:00
Chris Lu 0d5683fb0e todo: load path-specific store from filer.toml 2020-12-18 03:05:42 -08:00
Chris Lu 3269fd7eaf filer: use store by path 2020-12-18 02:57:49 -08:00
Chris Lu e605f1e001 refactor, add getActualStore() by path 2020-12-18 02:55:00 -08:00
Chris Lu 23903aa95b refactoring 2020-12-18 02:35:45 -08:00
Chris Lu 4e6b316913 assert FilerStoreWrapper is VirtualFilerStore 2020-12-18 02:17:06 -08:00
Chris Lu 1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu c45d6a865b Merge branch 'master' into support_ssd_volume 2020-12-13 22:14:18 -08:00
Chris Lu 38fc89d041 filer store: sql update if any insert error happens 
fix https://github.com/chrislusf/seaweedfs/issues/1673
 2020-12-13 20:49:44 -08:00
Chris Lu 23280257df sql put kv: avoid unnecessary update 2020-12-13 19:45:47 -08:00
Chris Lu 0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu 14910d035c the pagination size was too big for recursive deletion 2020-12-12 13:26:10 -08:00
Chris Lu 03637d6f57 s3: move "delete-directory-if-empty" to read time 
move "delete-directory-if-empty" to read time instead of entry deletion time

the listing speed for a s3 bucket folder will slow down if it has many sub folders

related to 0d345ac97d

fix https://github.com/chrislusf/seaweedfs/issues/1647

fix https://github.com/chrislusf/seaweedfs/issues/1670
 2020-12-12 03:38:34 -08:00
Chris Lu 37075a414d adjust logs 2020-12-12 03:33:57 -08:00
Chris Lu 3fedfec1e7 check cross device rename error 2020-12-10 23:50:32 -08:00
Chris Lu c2f18a10cb minor 2020-12-10 22:23:22 -08:00
Chris Lu 4b0c2a846b add logs 2020-12-10 19:55:28 -08:00
Chris Lu 83078ac6ce filer: change to /etc/seaweedfs folder on filer 
fix https://github.com/chrislusf/seaweedfs/issues/1666
 2020-12-10 11:11:02 -08:00
Chris Lu 765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu a9c619c270 added error handling for saving 2020-12-09 01:34:24 -08:00
Chris Lu 8e78187a97 add back last read chunk cache to reader and properly close the reader 2020-12-08 22:26:46 -08:00
Chris Lu 900d22c6ec mount: avoid memory leaking read buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1654

the reader goes together with the file handle, which may stay for a long time.
 2020-12-08 02:38:53 -08:00
Chris Lu fce8803087 break import cycle 2020-12-07 00:29:17 -08:00
Chris Lu 01e2da5782 refactoring 2020-12-06 23:16:20 -08:00
Chris Lu 9acda432fe fix import cycle 2020-12-06 20:12:52 -08:00
Chris Lu 5931a2f53f filer conf: support json conf only 2020-12-06 19:47:06 -08:00
Chris Lu f4abd01adf filer: cache small file to filer store 2020-11-30 04:34:04 -08:00
Chris Lu a9c6be5fc3 set creation time 2020-11-30 03:11:52 -08:00
Chris Lu 0d345ac97d s3: remove empty parent folder on delete 
fix https://github.com/chrislusf/seaweedfs/issues/1637
 2020-11-26 12:21:58 -08:00
Chris Lu cc2bd97ad9 refactor 2020-11-26 11:25:56 -08:00
Chris Lu 0da7ecfd29 go fmt 2020-11-26 11:22:30 -08:00
Chris Lu effa00ed08 refactor 2020-11-26 11:21:58 -08:00
Chris Lu 513bcd6e0d filer: avoid duplicated FindEntry for deletion 2020-11-26 11:14:56 -08:00
Chris Lu eab53ea80d filer leveldb store: a bit more efficient directory listing with prefix 2020-11-22 21:10:41 -08:00
Chris Lu 290b5e2cd0 directly delete file chunks 
keeping current async deletions for now
 2020-11-17 17:20:21 -08:00
Chris Lu dc304342b2 fs.configure: configurable volume growth 2020-11-17 01:00:02 -08:00
Chris Lu 5f19e81dab filer confi: support hierachical configuration 2020-11-16 16:50:12 -08:00
Chris Lu c0d279c54e filere.conf: prefer to use json format 2020-11-15 21:48:17 -08:00
Chris Lu ee2fa14dbe filer conf: delete location specific configuration 2020-11-15 20:15:47 -08:00
Chris Lu 71056dae07 fs.configure: read and local add filer configuration 2020-11-15 18:09:35 -08:00
Chris Lu 0ea5c087ce go fmt 2020-11-15 16:59:28 -08:00
Chris Lu 95c0de285d refactoring 2020-11-15 16:58:48 -08:00
Chris Lu 500bcab953 refactoring 2020-11-15 14:41:56 -08:00
Chris Lu 590f02179d filer: load filer conf when starting 2020-11-15 14:06:03 -08:00
Chris Lu 0a406f652e load filer conf and match by prefix 2020-11-15 00:26:05 -08:00
Chris Lu 0fc9ffc603 able to read filer.conf if updated 2020-11-14 21:21:20 -08:00
Chris Lu 442e092995 filer watch "/etc" folder for configuration changes 2020-11-14 14:26:08 -08:00
Konstantin Lebedev aa30604cec fix tests 2020-11-12 08:49:38 +05:00
Konstantin Lebedev 1eec5c8d5d gen pb 2020-11-12 04:10:06 +05:00
Konstantin Lebedev fc7baef5bb fiil serverUrls sorted by data center 2020-11-12 02:13:33 +05:00
Chris Lu 8750cac090 move to util.RetryWaitTime 2020-11-01 02:36:43 -08:00
Chris Lu df8d976bb0 refactoring 2020-11-01 01:58:48 -07:00
Chris Lu 16fae84414 fix format error 2020-10-24 20:11:31 -07:00
Chris Lu e9d40b80b2 less verbose logs 2020-10-24 09:42:54 -07:00
Chris Lu b81359823f postgres: support empty user 2020-10-22 14:27:47 -07:00
Chris Lu d75bc62196 to read files that are quickly changing 2020-10-22 12:59:15 -07:00
Chris Lu 5179e559f7 skip empty logs 2020-10-22 00:35:48 -07:00
Chris Lu bd103c143a add lock for vidCache 2020-10-21 19:28:59 -07:00